[fjdiazg]

I have created a Visual Basic application, yes I'm that old, that takes all the PID files in a directory and consolidates into one big file, discards the empty files, it adds a date column at the beginning, so far it account for dates changes but not for month/year changes (work in progress)

If someone is interested in testing it, please let me now by private message.

Thanks,

 

[colchiro]

Maybe if you described what you're trying to do, you'd have more interest?

 

[fjdiazg]

Maybe if you described what you're trying to do, you'd have more interest?

Thanks for the suggestion,

This idea was fired by the long outage on OnStar and my volt.com, I wanted to have statistics from the usage of my volt and I have been using mygreenvolt for a while, so as it is cumbersome to add all those files, keep the changes in date and consolidate the information, i created the application.

So far it can consolidate your files into a big one, and wil take care of simple date changes within a file, it does not count for changes in months or years yet.

I got a request for testing and my original plan was to release a fully date change aware version, but for different reasons I had to put the development in the back burner.

I hope I can offer that version soon.

Freddy

 

[colchiro]

...and what does "testing" involve?

 

[fjdiazg]

...and what does "testing" involve?

Oh, Just run the app with your data and verify it works properly, I do my own testing, but it's always good to have someone test and see if there are conditions that make the application fail...

Thanks,

 

[canehdian]

It sounds like this could be accomplished far easier using off-the-shelf products.
Microsoft has come a long way in terms of handling data - using the latest version of Excel or PowerBI you can ingest an entire folder of CSVs, merging into one, applying column formats and other step-by-step import manipulations
And then throw it into tables or visualizations to see your data output.

It takes what would normally be a painstaking process or complex macro and makes it far simpler.

 

[fjdiazg]

It sounds like this could be accomplished far easier using off-the-shelf products.
Microsoft has come a long way in terms of handling data - using the latest version of Excel or PowerBI you can ingest an entire folder of CSVs, merging into one, applying column formats and other step-by-step import manipulations
And then throw it into tables or visualizations to see your data output.

It takes what would normally be a painstaking process or complex macro and makes it far simpler.

That sounds interesting, and will save me from a lot of coding, I will research on this.

 

[canehdian]

Using excel 2016, you would start a new powerquery with source from text/folder (I believe it may be available as an add-on for earlier versions)

Have all of your source CSVs in the same folder, no other files that might confuse the auto loading

Then choose combine and load, done. If you want advanced edits to automatically come through, you can edit the query - for example change date formats, filter out repeated headers, delete columns or format them differently, add calculated columns, etc

Then if you add more files to the folder it's just a matter of hitting refresh and they're all brought in automatically and formatted in the exact steps you built the first time around.

I've yet to experiment if it's smart enough to match like named columns or if they all need to be identical (e.g. if you removed a column in future files would it just match up with the same headers and leave that one blank, or would it place data in the wrong columns or fail, etc)

 

[fjdiazg]

Using excel 2016, you would start a new powerquery with source from text/folder (I believe it may be available as an add-on for earlier versions)

Have all of your source CSVs in the same folder, no other files that might confuse the auto loading

Then choose load and combine and done. If you want advanced edits to automatically come through, you can edit the query - for example change date formats, filter out repeated headers, delete columns or format them differently, add calculated columns, etc

Then if you add more files to the folder it's just a matter of hitting refresh and they're all brought in automatically.

I've yet to experiment if it's smart enough to match like named columns or if they all need to be identical (e.g. if you removed a column in future files would it just match up with the same headers and leave that one blank, or would it place data in the wrong columns or fail, etc)

Cool! Thanks, now I need to upgrade excel because of this!

 

[canehdian]

Just did a quick test on that last statement there (need to know it for the day job anyway )

And it will fail if you delete a column (all data shifted over one) and adding a column will also make it mismatch.
Not used it enough to know if you can use advanced editing to account for it

Worst case you need to build a query for each file type (same columns in each group) and then merge those queries by mapping columns to ensure they line up appropriately. I'd have to play with it more to figure out alternatives.

If all of your files are identical columns, then no worries - it's all automatic.

 

[fjdiazg]

Just did a quick test on that last statement there (need to know it for the day job anyway )

And it will fail if you delete a column (all data shifted over one) and adding a column will also make it mismatch.
Not used it enough to know if you can use advanced editing to account for it

Worst case you need to build a query for each file type (same columns in each group) and then merge those queries by mapping columns to ensure they line up appropriately. I'd have to play with it more to figure out alternatives.

If all of your files are identical columns, then no worries - it's all automatic.

Good point!

Thanks,
Freddy