Mapping a collateral data pipeline
Light maintenance and cataloguing of a collateral data pipeline I did not build: an inventory and diagrams of its PySpark, Kafka and Django code, so the projects I was building for the same client could connect to it. Client anonymised.
- Context
- A financial operations suite across three repositories, with a Django backend, a Vue frontend and PySpark pipelines, built by another team before I arrived. The projects I was building for the same client had to read from and feed this suite, and there was no map of it.
- Pressure
- Connect my projects to a production system I did not write, without changing how it behaves, in a flow that runs across staging, pre-production and production.
- Decision
- Catalogue what already existed before writing any integration. In maintenance, change little, and only where the catalogue showed what the change would affect.
- Execution
- I read the three repositories and built an inventory of the Kafka topics and consumers per domain, the error queue (DLQ), the PySpark silver and gold layers and the S3 and SQS entry and exit points, with diagrams of an event's path from intake to report. The maintenance was light and targeted.
- Evidence
- The catalogue and the diagrams were the basis for integrating my projects with the suite.
- Impact
- My projects for this client talked to the suite from a map rather than by trial and error. The system and its architecture decisions belong to the original team; the client and the business rules stay out under confidentiality.