Skip to content

Commit

Permalink
docs: 🔧 elaborate forecast validation and add model market idea
Browse files Browse the repository at this point in the history
Adds more detail about prediction validation (target vs predicted errors), fixes typos (society), and proposes a competing AI model market with rewards based on prediction accuracy.
  • Loading branch information
davidgasquez committed Jan 11, 2025
1 parent 1f3a718 commit 1f7f14d
Show file tree
Hide file tree
Showing 3 changed files with 4 additions and 3 deletions.
2 changes: 1 addition & 1 deletion Datathons.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
6. Plot learning curves ([sklearn](https://scikit-learn.org/stable/modules/learning_curve.html) or [external tools](https://github.com/reiinakano/scikit-plot)) to avoid overfitting.
7. Plot real and predicted target distribution to see how well your model understand the underlying distribution. Apply any postprocessing that might fix small things.
8. Tune hyper-parameters once you've settled on an specific approach ([hyperopt](target distribution), [optuna](https://optuna.readthedocs.io/)).
9. Plot and visualize the predictions (histograms, random prediction, ...) to make sure they're doing as expected. Explain the predictions with [SHAP](https://github.com/slundberg/shap).
9. Plot and visualize the predictions (target vs predicted errors, histograms, random prediction, ...) to make sure they're doing as expected. Explain the predictions with [SHAP](https://github.com/slundberg/shap).
10. Think about what postprocessing heuristics can be done to improve or correct predictions.
11. [Stack](https://scikit-learn.org/stable/auto_examples/ensemble/plot_stack_predictors.html) classifiers ([example](https://www.kaggle.com/couyang/featuretools-sklearn-pipeline#ML-Pipeline)).
12. Try AutoML models. For tabular data: [TPOT](https://github.com/EpistasisLab/tpot), [AutoSklearn](https://github.com/automl/auto-sklearn), [AutoGluon](https://auto.gluon.ai/stable/index.html), Google AI Platform, [PyCaret](https://github.com/pycaret/pycaret), [Fast.ai](https://docs.fast.ai/), [Alex](https://github.com/Alex-Lekov/AutoML_Alex).For time series: [AtsPy](https://github.com/firmai/atspy), [DeepAR](https://docs.aws.amazon.com/forecast/latest/dg/aws-forecast-recipe-deeparplus.html), [Nixtla's NBEATS](https://nixtlaverse.nixtla.io/neuralforecast/models.nbeats.html), [AutoTS](https://github.com/winedarksea/AutoTS).
Expand Down
3 changes: 2 additions & 1 deletion Future.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ History teaches us that in 100 years from now [[Openness|some of the assumptions
- Give birth without advanced assistance.
- Not caring for all the [animal suffering in the wild](https://longtermrisk.org/the-importance-of-wild-animal-suffering/).
- Nature is not safe! The default is suffering. The current mentality is that nature is good and disruptions from nature are bad.
- The ignorance of Social Media and its [full impact on s_o_ciety](https://twitter.com/M_B_Petersen/status/1483457679800651787).
- The ignorance of Social Media and its [full impact on society](https://twitter.com/M_B_Petersen/status/1483457679800651787).
- Is "being bad for society" an emergent property of social networks as they grow?
- Current Voting Systems.
- Not relying more into tools like Prediction Markets (e.g: [to spot papers that might not replicate](https://vitalik.eth.limo/general/2024/11/09/infofinance.html)).
Expand Down Expand Up @@ -41,3 +41,4 @@ History teaches us that in 100 years from now [[Openness|some of the assumptions
- Prolly/Merkle Trees
- Differential/Timely Dataflow
- Zero-Knowledge Proofs
- Have a market of ML/AI models each competing to predict an outcome. When the outcome actually happens, rank them on a leaderboard by how close their predictions were and reward accordingly.
2 changes: 1 addition & 1 deletion Time.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ Time is the most valuable and least replaceable resource. Just like money, [time
- Do the tasks in the right order! One strategy might be starting with the one that could make the others irrelevant or easier.
- Time-boxing is [[Planning]] how you spend your days in advance and it's so effective because it allows you to iterate. If you didn't complete everything you outlined, you know exactly why -- because you've documented how you planned to spend your time.
- You'll never have any more time. You have, and have always had, all the time there is. [How you spend your time is a choice](https://leebyron.com/4000/).
- Success can be measured in the percentage of time you have under your control. [To achieve success](https://blog.samaltman.com/how-to-be-successful):
- Success can be measured in the percentage of time you have under your control:
- Compound yourself. Compounding is magic. Keep long-term thinking with a broad view of how different systems in the world are going to come together.
- Learn to think independently.
- Make it easy to take risks. Most people overestimate risk and underestimate reward. It's often easier to take risks early in your career; you don't have much to lose, and you potentially have a lot to gain.
Expand Down

0 comments on commit 1f7f14d

Please sign in to comment.