Types of Data
How I think about types of data.
<h2>Batch Data</h2> <ol><li><p>Single Entities</p></li><li><p>Sets of Data</p></li></ol> <h2>Streaming Data</h2> <p>Streaming Data is a never ending flow of data. Some examples include Twitter tweets, hospital bed telemetry data, subway ticket scans, etc. I like to further categorize data seen as these:</p> <ol><li><p>Application Logs</p></li><li><p>Intra-application Messages</p></li><li><p>Inter-application Messages</p></li><li><p>Business Events</p></li></ol> <h3>Application Logs</h3> <ul><li><p>In this scenario, your application is writing logs so engineers can see what's going on and troubleshoot technical issues.</p></li><li><p>This data is meant for you and only you. It's not meant for business folks to use so avoid providing it for analytical purposes.</p></li><li><p>Ensure you log discrete data, avoid logging sensitive data, and use appropriate logging levels.</p></li><li><p>Educate others on how to find your application's logs and what IDs can tie logs across applications.</p></li></ul> <h3>Intra-application Messages</h3> <ul><li><p>In this scenario, your application has separate components that communicate with one another.</p></li><li><p>This data is meant for you and only you. It's not meant for business folks to use so avoid providing it for analytical purposes.</p></li></ul> <h3>Inter-application Messages</h3> <ul><li><p>In this scenario, your application submits data to another application you do not control.</p></li><li><p>This data is meant for the receiving system only. It's not meant for business folks to use so avoid providing it for analytical purposes.</p></li></ul> <h3>Business Events</h3> <ul><li><p>In this scenario, applications fire true fire-and-forget business events meant for others to consume, including subscribers responsible for populating the analytical data store.</p></li><li><p>Example events might include Order Placed, New Customer Registration, Promo Code Used, and so on. The data format is crafted specifically for others to consume.</p></li><li><p>Avoid making breaking changes to your events and if a breaking change is needed, version your event. Plus, give folks time to move to the newer version (i.e. publish both versions for some time). </p></li></ul> <p></p>