Fixed width format sample data and specifications

For testing purposes, you can find a fixed width test file here.

Advertisements

Json: great site for faking up test data

Great site here. First 2 show the template I wrote for my composer test data. Final one is the author’s show-it-all template.

Here I try to reduce the Json to the bare minimum (I just want the “composer” line). But I can’t get the 2 dates to work inside the function. Doubtless me. It will do for now.

[
 {
 'repeat(3,3)': {
 dateOfBirth: '{{moment(this.date(new Date(1684, 0, 1), new Date(1722, 0, 1))).format("L")}}',
 dateOfDeath: '{{moment(this.date(new Date(1722, 0, 2), new Date(1801, 0, 1))).format("L")}}', 
 composer: function (tags) {
 return tags.surname().toUpperCase() + ', ' + tags.firstName() + 
 ' (' + this.dateOfBirth + ' - ' + this.dateOfDeath + ') ' ;
 }
 }
 }
]

PowerShell: generating test data

# Create a set of fixed width records purely for volume testing purposes. The unique key part
# is generated for each record, and the remainder of the record is repeated every time. In the
# constants below, the $COMPOSER_RECORD_LENGTH is inclusive of $KEY_LENGTH.
# Note that the creation of records using StreamWriter is very much faster than using Out-File:
# at some point I'll get around to measuring that. But for now, 0.5 million records on an I3 with
# SSD takes about 1 minute.

# Edit below here as required...
$RECORDS_TO_CREATE = 500000
$DEBUG_FREQUENCY = 5000
$OUTPUT_FILE = 'C:\temp\BulkComposers.txt'
$KEY_LENGTH = 50
$RECORD_REMAINDER = "Mozart Wolfgang Amadeus Leopold Salzburg Vienna 1756 1791 "
$COMPOSER_RECORD_LENGTH = 1092
# Edit above here as required...

$NEWLINE = "`r`n"
$recordRemainderPadded = $RECORD_REMAINDER.PadRight($($COMPOSER_RECORD_LENGTH-$KEY_LENGTH))

$mode = [System.IO.FileMode]::Create
$access = [System.IO.FileAccess]::Write
$sharing = [System.IO.FileShare]::Read
$encoding = [System.Text.UTF8Encoding]
$bufferSize = 4096

$fs = New-Object System.IO.FileStream($OUTPUT_FILE, $mode, $access, $sharing, $bufferSize, $encoding)
$stream = New-Object System.IO.StreamWriter($fs)

"Starting creation of $($RECORDS_TO_CREATE) composer records"

1..$RECORDS_TO_CREATE | foreach {
 $composerId = [guid]::NewGuid().ToString().PadRight(50)
 $composerRecord = $composerId + $recordRemainderPadded
 $stream.WriteLine($composerRecord, [System.Text.UTF8Encoding])

if ($_%$DEBUG_FREQUENCY -eq 0) {
 "$(get-date): Creating record $_"
 }
}

"Completed creation of $($RECORDS_TO_CREATE) composer records"
"See file [$OUTPUT_FILE]"

$stream.Dispose()
$fs.Dispose()
$stream.Close()
$fs.Close()