We are looking for a skilled Big Data developer to join our startup, and build a cutting-edge platform that leverages AI-driven political narratives to predict the performance of niche stocks (e.g., defense stocks under Republican administrations). You will design and implement a scalable back-end data infrastructure to handle the ingestion, storage, query and processing of historical and real-time data at scale. The system will support user-generated AI trader personas, backtest strategies, simulate performance, and scale up testing by thousands of users. 30-40+ Stocks across multi-year datasets.

Key responsibilities

1. Data ingestion and pipelines:
   • Created ETL pipelines to ingest data from financial APIs (e.g., yfinance, Quandl), Google Trends (via pytrends), and X/Twitter APIs.
   • Manage large data sets (e.g., terabytes of historical stock prices, real-time updates, unstructured sentiment data).
   • Use orchestration tools (e.g., Apache Airflow, Luigi) to automatically update data daily/hourly.
   • Ensure data quality through cleaning, data deduplication, and handling of missing values.
2. Database design and management:
   • Migrate from SQLite to a scalable big data solution (e.g., AWS S3 + Athena, Google BigQuery, Hadoop/HDFS with Hive).
   • Implement caching (e.g., Redis and pandas DataFrames) for recurring queries and backtesting.
   • Enable complex queries (for example, “SELECT Average Defense Stock Returns Where President = ‘Trump’ and date between ‘2017-01-20’ and ‘2021-01-20’”).
   • Improve performance with efficient indexing, partitioning (by date/inventory), and joins.
3. Engine testing and processing:
   • Integrate Python libraries (e.g. backtrader, backtesting.py) to simulate the strategy.
   • Scale computations using distributed processing (e.g., Apache Spark, Dusk) for parallel backtesting across large datasets.
   • Support for personalized AI inquiries (e.g., “Test actor’s narration about oil stocks during the Trump presidency”).