TST/PERF: make a test input creater instead of np.random.randn mainly for perf testing

[jreback]

to allow easy testing of c-contig/f-contig and long frames etc

see #3089
http://mail.python.org/pipermail/pandas-dev/2013-March/000008.html

I expected np.array(order='C') , np.array(order='F') to work, it doesn't.

[jreback]

I think you have to do

np.array(np.random.rand(10,2),order='C')

[stephenwlin]

we should also exercise non-contiguous cases, too, because those result from slicing larger contiguous arrays.

[jreback]

absolutely, if you have time, maybe put together of a list of things we shoudl test (on perf basis) where these things matter?

[stephenwlin]

well, that's all i can think of now (obviously I didn't think of this before it came up as a perf issue, otherwise I would have tested it already), but I'll make note if I think of any more

It looks like the perf regression was 50% - but on a 1ms test.
Just wondering what scale of data are we talking about here where
this would seriously impact users?

[stephenwlin]

@y-p take is a primitive operation for a lot of other operations though...I think the test suite is not representative of real world usage in general (and performance is definitely noticeable for large data sets...like >1GB, which I've actually had to deal with for personal usage already) so it's very hard to tell what matters and what doesn't; it's like reading tea leaves

of course, I guess i'm trying to understand what regression we're talking about, since
jeff's original vbench that prompted this only showed a couple of reindex_foo outliers.

Basically, you've landed a big optimization by taking array order into account,
it has nothing to do with the original regression. did I get that right?

[stephenwlin]

do you mean the latest PR, #3130? that does address the regression in reindex_frame_level_align and reindex_frame_level_reindex, by turning off the original optimization for small row/column sizes, and it happens to optimize other cases as well (by allowing the optimization in cases where it could be used but isn't) but I don't know how well those are tested by the suite...we should move comments on this over to #3130 if that's what you're talking about (the regression was only 25% on my machine, due to 32-bit/64-bit differences, so that PR undoes it)

[stephenwlin]

I'm trying to test a further optimization refinement (described in the FINAL EDIT portion of this SO question) and I'm getting some extreme variability in my results (with or without the new BUILD_CACHE_DIR, although it seems to be worse with it...still not sure if there's something going on with that or if it's just a fluke).

I've talked to some performance experts in the C++ SO chatroom about this and the consensus seems to be that randomly generating a set of indices once but reusing them over and over for multiple runs of the same test (as is the case for some, but not all, of the suite tests...I believe this is the case at least... can anyone confirm that the setup portions of the tests are only executed once rather than repeatedly?) is leading to inconsistent results from run to run because of the effect of the layout on caching and alignment issues.

To get more consistent results, I think we need to do one of the following:

• Always generate new random indices per iteration of each test so that the variability gets averaged out
• Use sets of indices hardcoded into the test suite for each test

The first is probably better but it means that the time of the random index generation is part of the results of the test, which is non-ideal. The second might lead to less coverage in case hardcoded indices don't adequately cover real use cases.

Not 100% sure how much of an issue this is, but from the results I've been seeing it seems like this is the most likely culprit...still investigating (a little bit hard because of test turnaround time).

Does vb actually do multiple iterations and averaging? I didn't know.

So you're suggesting that adding the default random seed to test_perf will actually
make variability worse?

If you can post a script example for me to verify that what you suggests eliminates variation
on my machine as well, that'd be great.

[stephenwlin]

@y-p I don't know either about vb_suite...I'm guessing the setup section is done once and all the multiple iterations are of the statement passed as the first argument to Benchmark():

i.e.

groupby_first = Benchmark('data.groupby(labels).first()', setup,
                          start_date=datetime(2012, 5, 1))

calls setup once and then iterates 'data.groupby(labels).first()' repeatedly.

In this case, setup creates a random permutation of labels:

labels = labels.take(np.random.permutation(len(labels)))

but the same labels are (presumably) used repeatedly so any performance effect of structural features of the permutation will be present in every iteration rather than averaged out.

As per your statement

So you're suggesting that adding the default random seed to test_perf will actually
make variability worse?

I think I'm suggesting the opposite actually (unless I'm misunderstanding you)...if we fix the seed (it's not fixed right now, is it?), which I didn't actually consider, then every run of vb_suite will be using the same random numbers so we'll eliminate the effect of variability entirely in theory (but possibly the test will be non-representative by accident). Or we could go the opposite extreme, and make every single iteration of each test variable, so the variability hopefully averages out. What doesn't work is the middle ground where we non-deterministically generate a random set of numbers once and reuse them over and over (which I presume is what's happening right now).