Complete assignment 1: Parquet feature engineering

[alf-99]

• Load price data from parquet files
• Create lag features, returns, and price ranges
• Convert to pandas and calculate moving averages
• Answer written analysis questions

What changes are you trying to make? (e.g. Adding or removing code, refactoring existing code, adding reports)

I completed Assignment 1 on working with parquet files and feature engineering. The main changes were:
-Loading stock price data from parquet files using Dask.
-Creating new features: 1-day lags for Close and Adj Close, daily returns, and daily price ranges.
-Converting the data to pandas to calculate 10-day moving averages of returns.
-Answering questions about why pandas was needed for the moving average.

What did you learn from the changes you have made?

-How to work with parquet files stored in nested directories (ticker folders).
-The difference between Dask (for big data) and pandas (for in-memory data).
-Why feature engineering like lags and returns is important for time-series data.
-That Dask has some limitations with complex grouped rolling operations.

Was there another approach you were thinking about making? If so, what approach(es) were you thinking of?

I thought about trying to do the moving average in Dask instead of converting to pandas, but the rolling() method with groupby() kept giving me issues. I also considered calculating more features like volatility or additional lag periods, but stuck with the assignment requirements.

Were there any challenges? If so, what issue(s) did you face? How did you overcome it?

The main challenge was finding the parquet files - they were in ticker subdirectories, not all in one folder. I had to use recursive=True with glob() to find them all. Also, Windows file permissions caused issues when running the data engineering notebook, so I had to skip saving the features data.

How were these changes tested?

-Verified parquet files were found.
-Checked that new columns were created correctly (Close_lag_1, Returns, hi_lo_range).
-Ran all code cells from top to bottom to ensure no errors

A reference to a related issue in your repository (if applicable)

N/A - This is Assignment 1 submission.

Checklist

• [x ] I can confirm that my changes are working as intended.

[fyellow]

Excellent work!

Everything looks good! No changes are needed. You do have a warning (which shouldn't be actually effecting your results), but I believe to get rid of it, you have to specify "meta" which is the order of the columns and their data type during the grouped shift operations using Dask.