![]() ![]() Exploratory analysis involves time and hopefully you've arranged some time because data can have much more context when given over time. This can be a good time to go back to your client or project manager to discuss what it is you're seeing and make sure that it's aligned with your expectations. When you have less knowledge about the data, you tread very lightly in your expectations and watch it carefully, documenting observations and consistencies over time. Or if it’s associated with production data and needs that context, perhaps you can create a separate environment where they integrate. If you’re in an exploratory phase, you may want to segregate that data in another environment where it won’t have to play with production data. What is of utmost importance is to bring that data in for some basic and exploratory analysis. That doesn’t mean that you won’t start ingesting it. If it’s the latter, more investigation is needed. It could also be a file that your business is not quite sure what it will do with yet. The data you receive can be an important file that is extremely integral to your business. You do NOT want your company to be that one. What they don’t want is a company suing them because of data that just went in willy-nilly and caused all kinds of havoc in their systems. What they will appreciate are the clients that notify them of issues right away because those clients validate the data BEFORE ingestion. Issues arise even from the best and most organized of companies. Their salaries depend on it.īut as someone who worked with the data from one of these companies for years, don’t think that everything is going to come up roses. ![]() It can be very nice to work with a third-party company whose whole business is making sure you’re getting accurate and timely data in the format that they guarantee. They want you to use it, so they'll give you a giant document with definitions and layouts and you can get started right away. ![]() They care what it looks like and how it’s formatted. For businesses that sell data, the data is their product. My preferred data to ingest, if I had my dream world of data ingestion (queue the birds and dancing data strolling through a nice meadow), would be from a third-party system that specializes in data extraction. Once you've worked through the questions of what data it is you will be receiving, it's time to think about the technical horizon for the data. In this second part of the Art of Data Ingestion, we'll discuss technical considerations for ingesting data. In part one of this series, we discussed how to prepare for the ingestion of data. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |