TL;DR
Board currently provides no overview of which Data Readers pull from which source tables/views/files. This creates avoidable risk and slows down troubleshooting, onboarding, and impact analysis. I’m proposing a built‑in Data Reader Source Catalog that shows all Readers, their source objects, targets, and last‑run status in one searchable view. This would dramatically improve governance, transparency, and operational efficiency for anyone maintaining Data Models.
Problem statement
Today there is no single overview of which Data Readers load which source tables/queries, into which Entities/Cubes, under which environments, and with what schedule/status. This creates avoidable risk and effort for admins and developers:
- Troubleshooting failures requires manual hunting across Readers and Procedures.
- Impact analysis for schema changes (e.g., column rename, source view replacement) is slow and error‑prone.
- Audit, compliance, and handovers are harder without up‑to‑date documentation.
- New team members lack visibility into the integration footprint of a Data Model.
This feature strengthens governance and supports Board providing enterprise‑grade transparency across the data pipeline.
Who is affected
- Data model designers/admins maintaining Readers/Procedures across DEV/TEST/PROD
- Support teams diagnosing failed loads and rejected rows
- Data owners announcing upstream schema changes and needing a fast impact check
Current workarounds (which fall short over time)
- Naming conventions and spreadsheets drift out of date
- Reading individual Reader configurations is time‑consuming
- Logs help after failures, not for proactive oversight
Proposed solution
Introduce a Data Reader Source Catalog in Data Model > Data Readers with:
1. Tabular catalog
Columns such as:
- Reader Name
- Source Type (SQL/Text/SAP)
- Connection
- Source objects (table(s)/view(s)/file path(s)/query name)
- Targets (Entities, Cubes)
- Procedure(s) using this Reader
- Schedule
- Last run result (time, duration, records read/validated/rejected)
- Environment (DEV/TEST/PROD)
- Owner
2. Trace & impact
- Click a Reader to see all mapped source objects and all targets with a many‑to‑many lineage diagram (Source → Reader → Entity/Cube).
- Search by source pattern (e.g.,
%vw_Sales%) to instantly list all Readers impacted by an upstream change.
3. Export
- Export the catalog to CSV/JSON for documentation, governance, and compliance.
Benefits
- Faster impact analysis: When a DBA renames
vw_SalesOrders, find every Reader that references it in seconds. - Faster troubleshooting: From a failed load, jump from the Reader to impacted targets and related Procedures.
- Better governance: Exportable inventory of “what loads what” improves audits and handovers.
- Onboarding speed: New developers can understand the data ingestion landscape at a glance.
- Preventive maintenance: Spot orphan Readers, dead paths, and high‑reject protocols.
**If you made it this far, this idea probably resonates with you. Please upvote it, share your pain points in the comments, and tell colleagues who might benefit**
