Unclear How to Save S3 to Offset

[cjhanks]

It is very unclear how precisely how one is intended to use the HTTP client to save into a file at some non-0 offset when using the 'SetResponseStreamFactory'.

Specifically, I have a large sparse file (let's say 40gb+) which I would like to fill on-demand. The only examples I can find for dealing with streams are in https://github.com/awslabs/aws-sdk-cpp/blob/master/aws-cpp-sdk-s3-integration-tests/BucketAndObjectOperationTest.cpp and there doesn't appear to be anything near what I need.

In the same train of thought, this nebulous comment kind of worries me... should it?

// The 'truncate' is required to ensure that if the file download IS successful, then it can be written to the
// FStream (if ::trunc is not specified, then the FStream.write fails for some unknown reason)

[JonathanHenson]

You can return anything that inherits from std::iostream in your closure, we haven't implemented something that does what you need, but if you want to do that, you can create some sort of adapter to make it happen.

As for that comment, if the http response returns an error, we write the payload out to the stream, but if the in flag is not turned on, we can't parse the error. Nothing will break if you don't pass that flag, but you will recieve an "UNKNOWN" error if something goes wrong.

[JonathanHenson]

One thing you could do is create your file stream in the closure, and then seek it to where you want it to start. The system should start writing at the seeked position.

[cjhanks]

And I should not expect any internal retry mechanisms to conflict with this logic?

[JonathanHenson]

You can turn retries off or override the retry strategy if you are worried about it. But no, the response stream is recreated for each new request/response cycle (so a retry would just recreate the stream)

[JonathanHenson]

One word of caution. This is a use case I haven't thought about very much and in the event of an error coming across the wire, the error will be written to the stream you provide, so you'll either need to handle that, or possibly maintain your file io stuff separately and just use our default response stream, then write the response to your file manually after you've checked the error code.

I'm not to worried about that sort of thing for normal file stream use cases, but yours is special and as a result my be problematic if an xml doc error payload is in the middle of your 40gb file.

[JonathanHenson]

S3 also has a torrent api, just fyi

http://docs.aws.amazon.com/AmazonS3/latest/dev/S3Torrent.html

[cjhanks]

Okay, looks like I will steer clear of the untrodden path for now and take the hit from the disk copy. Thanks.

[JonathanHenson]

You don't have to do a disk copy, if you have sufficient ram available, the default stream will use a stringbuffer, but you need to be aware of how large the objects are that you are downloading.

[cjhanks]

So I tried to implement this, and it appears to be segfaulting.

#include <gtest/gtest.h>
#include <glog/logging.h>

#include <fstream>
#include <memory>

#include "rcp_net/NetFile.hpp"

#include <aws/core/client/ClientConfiguration.h>
#include <aws/core/http/HttpClientFactory.h>
#include <aws/core/http/HttpClient.h>
#include <aws/core/http/HttpRequest.h>
#include <aws/core/http/HttpResponse.h>

using namespace Aws;
using namespace Client;
using namespace Http;


Aws::Client::ClientConfiguration Config;

TEST(NET, Basic) {
    std::shared_ptr<HttpClientFactory> factory(new HttpClientFactory);
    std::shared_ptr<HttpClient> client(factory->CreateHttpClient(Config)); // sigh

    std::string filePath("someTestPath");
    size_t offset = 10;

    std::shared_ptr<HttpRequest> request(
        factory->CreateHttpRequest(
            Aws::String("http://google.com"),
            HttpMethod::HTTP_GET,
            [&filePath, &offset]() -> Aws::IOStream* {
                constexpr auto OpenFlag = std::ios_base::out | std::ios_base::binary;
                Aws::IOStream* ofs = new Aws::FStream(filePath, OpenFlag);
                CHECK(ofs->good());
                return ofs;
            }));

    std::shared_ptr<HttpResponse> response = client->MakeRequest(*request); //<-- *** Segfaults there ***
    ASSERT_EQ(HttpResponseCode::OK,
              response->GetResponseCode());

$> valgrind --tool=memcheck ./gtests --gtest_filter='NET.*'
==10845== Memcheck, a memory error detector
==10845== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==10845== Using Valgrind-3.10.0.SVN and LibVEX; rerun with -h for copyright info
==10845== Command: ./gtests --gtest_filter=NET.*
==10845== 
Note: Google Test filter = NET.*
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from NET
[ RUN      ] NET.Basic
==10845== Invalid read of size 8
==10845==    at 0x64F1309: std::ostream::sentry::sentry(std::ostream&) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845==    by 0x64F16BD: std::ostream::write(char const*, long) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845==    by 0x59089FB: Aws::Http::CurlHttpClient::WriteData(char*, unsigned long, unsigned long, void*) (in /usr/lib/libaws-cpp-sdk-core.so)
==10845==    by 0x7E2DCCF: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E474AF: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E41CD6: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E4B9DB: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E4C180: curl_multi_perform (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E437B2: curl_easy_perform (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x59098E8: Aws::Http::CurlHttpClient::MakeRequest(Aws::Http::HttpRequest&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const (in /usr/lib/libaws-cpp-sdk-core.so)
==10845==    by 0x4354C3: NET_Basic_Test::TestBody() (test_net.cpp:45)
==10845==    by 0x46E252: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (gtest.cc:2090)
==10845==  Address 0xe570098 is 24 bytes before a block of size 8,192 alloc'd
==10845==    at 0x4C2B800: operator new[](unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==10845==    by 0x64D0497: std::basic_filebuf<char, std::char_traits<char> >::_M_allocate_internal_buffer() (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845==    by 0x64D4541: std::basic_filebuf<char, std::char_traits<char> >::open(char const*, std::_Ios_Openmode) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845==    by 0x43511B: std::_Function_handler<std::iostream* (), NET_Basic_Test::TestBody()::{lambda()#1}>::_M_invoke(std::_Any_data const&) (fstream:281)
==10845==    by 0x5911E40: Aws::Utils::Stream::ResponseStream::ResponseStream(std::function<std::iostream* ()> const&) (in /usr/lib/libaws-cpp-sdk-core.so)
==10845==    by 0x5909745: Aws::Http::CurlHttpClient::MakeRequest(Aws::Http::HttpRequest&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const (in /usr/lib/libaws-cpp-sdk-core.so)
==10845==    by 0x4354C3: NET_Basic_Test::TestBody() (test_net.cpp:45)
==10845==    by 0x46E252: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (gtest.cc:2090)
==10845==    by 0x466E66: testing::Test::Run() (gtest.cc:2162)
==10845==    by 0x466F0D: testing::TestInfo::Run() (gtest.cc:2338)
==10845==    by 0x467014: testing::TestCase::Run() (gtest.cc:2445)
==10845==    by 0x4672B7: testing::internal::UnitTestImpl::RunAllTests() (gtest.cc:4243)
==10845== 
==10845== Invalid read of size 8
==10845==    at 0x64F132B: std::ostream::sentry::sentry(std::ostream&) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845==    by 0x64F16BD: std::ostream::write(char const*, long) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845==    by 0x59089FB: Aws::Http::CurlHttpClient::WriteData(char*, unsigned long, unsigned long, void*) (in /usr/lib/libaws-cpp-sdk-core.so)
==10845==    by 0x7E2DCCF: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E474AF: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E41CD6: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E4B9DB: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E4C180: curl_multi_perform (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E437B2: curl_easy_perform (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x59098E8: Aws::Http::CurlHttpClient::MakeRequest(Aws::Http::HttpRequest&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const (in /usr/lib/libaws-cpp-sdk-core.so)
==10845==    by 0x4354C3: NET_Basic_Test::TestBody() (test_net.cpp:45)
==10845==    by 0x46E252: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (gtest.cc:2090)
==10845==  Address 0xe570098

TEST(NET, Basic) {
    std::shared_ptr<HttpClientFactory> factory(new HttpClientFactory);
    std::shared_ptr<HttpClient> client(factory->CreateHttpClient(Config)); // sigh

    std::string filePath("someTestPath");
    size_t offset = 10;

    std::shared_ptr<HttpRequest> request(
        factory->CreateHttpRequest(
            Aws::String("http://google.com"),
            HttpMethod::HTTP_GET,
            [&filePath, &offset]() -> Aws::IOStream* {
                constexpr auto OpenFlag = std::ios_base::out | std::ios_base::binary;
                Aws::OFStream* ofs = new Aws::OFStream(filePath, OpenFlag);
                CHECK(ofs->good());
                return dynamic_cast<Aws::IOStream*>(ofs);
            }));


    std::shared_ptr<HttpResponse> response = client->MakeRequest(*request); //<-- *** Segfaults there ***
    ASSERT_EQ(HttpResponseCode::OK,
              response->GetResponseCode());

$> valgrind --tool=memcheck ./gtests --gtest_filter='NET.*'
==10845== Memcheck, a memory error detector
==10845== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==10845== Using Valgrind-3.10.0.SVN and LibVEX; rerun with -h for copyright info
==10845== Command: ./gtests --gtest_filter=NET.*
==10845== 
Note: Google Test filter = NET.*
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from NET
[ RUN      ] NET.Basic
==10845== Invalid read of size 8
==10845==    at 0x64F1309: std::ostream::sentry::sentry(std::ostream&) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845==    by 0x64F16BD: std::ostream::write(char const*, long) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845==    by 0x59089FB: Aws::Http::CurlHttpClient::WriteData(char*, unsigned long, unsigned long, void*) (in /usr/lib/libaws-cpp-sdk-core.so)
==10845==    by 0x7E2DCCF: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E474AF: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E41CD6: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E4B9DB: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E4C180: curl_multi_perform (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E437B2: curl_easy_perform (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x59098E8: Aws::Http::CurlHttpClient::MakeRequest(Aws::Http::HttpRequest&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const (in /usr/lib/libaws-cpp-sdk-core.so)
==10845==    by 0x4354C3: NET_Basic_Test::TestBody() (test_net.cpp:45)
==10845==    by 0x46E252: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (gtest.cc:2090)
==10845==  Address 0xe570098 is 24 bytes before a block of size 8,192 alloc'd
==10845==    at 0x4C2B800: operator new[](unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==10845==    by 0x64D0497: std::basic_filebuf<char, std::char_traits<char> >::_M_allocate_internal_buffer() (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845==    by 0x64D4541: std::basic_filebuf<char, std::char_traits<char> >::open(char const*, std::_Ios_Openmode) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845==    by 0x43511B: std::_Function_handler<std::iostream* (), NET_Basic_Test::TestBody()::{lambda()#1}>::_M_invoke(std::_Any_data const&) (fstream:281)
==10845==    by 0x5911E40: Aws::Utils::Stream::ResponseStream::ResponseStream(std::function<std::iostream* ()> const&) (in /usr/lib/libaws-cpp-sdk-core.so)
==10845==    by 0x5909745: Aws::Http::CurlHttpClient::MakeRequest(Aws::Http::HttpRequest&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const (in /usr/lib/libaws-cpp-sdk-core.so)
==10845==    by 0x4354C3: NET_Basic_Test::TestBody() (test_net.cpp:45)
==10845==    by 0x46E252: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (gtest.cc:2090)
==10845==    by 0x466E66: testing::Test::Run() (gtest.cc:2162)
==10845==    by 0x466F0D: testing::TestInfo::Run() (gtest.cc:2338)
==10845==    by 0x467014: testing::TestCase::Run() (gtest.cc:2445)
==10845==    by 0x4672B7: testing::internal::UnitTestImpl::RunAllTests() (gtest.cc:4243)
==10845== 
==10845== Invalid read of size 8
==10845==    at 0x64F132B: std::ostream::sentry::sentry(std::ostream&) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845==    by 0x64F16BD: std::ostream::write(char const*, long) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845==    by 0x59089FB: Aws::Http::CurlHttpClient::WriteData(char*, unsigned long, unsigned long, void*) (in /usr/lib/libaws-cpp-sdk-core.so)
==10845==    by 0x7E2DCCF: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E474AF: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E41CD6: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E4B9DB: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E4C180: curl_multi_perform (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E437B2: curl_easy_perform (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x59098E8: Aws::Http::CurlHttpClient::MakeRequest(Aws::Http::HttpRequest&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const (in /usr/lib/libaws-cpp-sdk-core.so)
==10845==    by 0x4354C3: NET_Basic_Test::TestBody() (test_net.cpp:45)
==10845==    by 0x46E252: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (gtest.cc:2090)
==10845==  Address 0xe570098 is 24 bytes before a block of size 8,192 alloc'd
==10845==    at 0x4C2B800: operator new[](unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==10845==    by 0x64D0497: std::basic_filebuf<char, std::char_traits<char> >::_M_allocate_internal_buffer() (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845==    by 0x64D4541: std::basic_filebuf<char, std::char_traits<char> >::open(char const*, std::_Ios_Openmode) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845==    by 0x43511B: std::_Function_handler<std::iostream* (), NET_Basic_Test::TestBody()::{lambda()#1}>::_M_invoke(std::_Any_data const&) (fstream:281)
==10845==    by 0x5911E40: Aws::Utils::Stream::ResponseStream::ResponseStream(std::function<std::iostream* ()> const&) (in /usr/lib/libaws-cpp-sdk-core.so)
==10845==    by 0x5909745: Aws::Http::CurlHttpClient::MakeRequest(Aws::Http::HttpRequest&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const (in /usr/lib/libaws-cpp-sdk-core.so)
==10845==    by 0x4354C3: NET_Basic_Test::TestBody() (test_net.cpp:45)
==10845==    by 0x46E252: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (gtest.cc:2090)
==10845==    by 0x466E66: testing::Test::Run() (gtest.cc:2162)
==10845==    by 0x466F0D: testing::TestInfo::Run() (gtest.cc:2338)
==10845==    by 0x467014: testing::TestCase::Run() (gtest.cc:2445)
==10845==    by 0x4672B7: testing::internal::UnitTestImpl::RunAllTests() (gtest.cc:4243)
==10845== 
==10845== Invalid read of size 8
==10845==    at 0x64F16D1: std::ostream::write(char const*, long) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845==    by 0x59089FB: Aws::Http::CurlHttpClient::WriteData(char*, unsigned long, unsigned long, void*) (in /usr/lib/libaws-cpp-sdk-core.so)
==10845==    by 0x7E2DCCF: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E474AF: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E41CD6: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E4B9DB: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E4C180: curl_multi_perform (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E437B2: curl_easy_perform (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x59098E8: Aws::Http::CurlHttpClient::MakeRequest(Aws::Http::HttpRequest&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const (in /usr/lib/libaws-cpp-sdk-core.so)
==10845==    by 0x4354C3: NET_Basic_Test::TestBody() (test_net.cpp:45)
==10845==    by 0x46E252: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (gtest.cc:2090)
==10845==    by 0x466E66: testing::Test::Run() (gtest.cc:2162)
==10845==  Address 0xe570098 is 24 bytes before a block of size 8,192 alloc'd
==10845==    at 0x4C2B800: operator new[](unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==10845==    by 0x64D0497: std::basic_filebuf<char, std::char_traits<char> >::_M_allocate_internal_buffer() (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845==    by 0x64D4541: std::basic_filebuf<char, std::char_traits<char> >::open(char const*, std::_Ios_Openmode) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845==    by 0x43511B: std::_Function_handler<std::iostream* (), NET_Basic_Test::TestBody()::{lambda()#1}>::_M_invoke(std::_Any_data const&) (fstream:281)
==10845==    by 0x5911E40: Aws::Utils::Stream::ResponseStream::ResponseStream(std::function<std::iostream* ()> const&) (in /usr/lib/libaws-cpp-sdk-core.so)
==10845==    by 0x5909745: Aws::Http::CurlHttpClient::MakeRequest(Aws::Http::HttpRequest&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const (in /usr/lib/libaws-cpp-sdk-core.so)
==10845==    by 0x4354C3: NET_Basic_Test::TestBody() (test_net.cpp:45)
==10845==    by 0x46E252: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (gtest.cc:2090)
==10845==    by 0x466E66: testing::Test::Run() (gtest.cc:2162)
==10845==    by 0x466F0D: testing::TestInfo::Run() (gtest.cc:2338)
==10845==    by 0x467014: testing::TestCase::Run() (gtest.cc:2445)
==10845==    by 0x4672B7: testing::internal::UnitTestImpl::RunAllTests() (gtest.cc:4243)
==10845== 
==10845== Jump to the invalid address stated on the next line
==10845==    at 0x8B48FFF9A57AE810: ???
==10845==    by 0x64F16E2: std::ostream::write(char const*, long) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845==    by 0x59089FB: Aws::Http::CurlHttpClient::WriteData(char*, unsigned long, unsigned long, void*) (in /usr/lib/libaws-cpp-sdk-core.so)
==10845==    by 0x7E2DCCF: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E474AF: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E41CD6: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E4B9DB: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E4C180: curl_multi_perform (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E437B2: curl_easy_perform (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x59098E8: Aws::Http::CurlHttpClient::MakeRequest(Aws::Http::HttpRequest&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const (in /usr/lib/libaws-cpp-sdk-core.so)
==10845==    by 0x4354C3: NET_Basic_Test::TestBody() (test_net.cpp:45)
==10845==    by 0x46E252: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (gtest.cc:2090)
==10845==  Address 0x8b48fff9a57ae810 is not stack'd, malloc'd or (recently) free'd
==10845== 
*** Aborted at 1443730215 (unix time) try "date -d @1443730215" if you are using GNU date ***
PC: @ 0x8b48fff9a57ae810 (unknown)
==10845== Syscall param msync(start) points to uninitialised byte(s)
==10845==    at 0x4E46B3D: ??? (syscall-template.S:81)
==10845==    by 0x7C02123: ??? (in /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1)
==10845==    by 0x7C02643: ??? (in /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1)
==10845==    by 0x7C04EAA: ??? (in /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1)
==10845==    by 0x7C06151: ??? (in /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1)
==10845==    by 0x7C064E8: ??? (in /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1)
==10845==    by 0x7C02A30: _ULx86_64_step (in /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1)
==10845==    by 0x5677442: google::GetStackTrace(void**, int, int) (in /usr/lib/x86_64-linux-gnu/libglog.so.0.0.0)
==10845==    by 0x567CB31: ??? (in /usr/lib/x86_64-linux-gnu/libglog.so.0.0.0)
==10845==    by 0x4E4733F: ??? (in /lib/x86_64-linux-gnu/libpthread-2.19.so)
==10845==    by 0x8B48FFF9A57AE80F: ???
==10845==    by 0x64F16E2: std::ostream::write(char const*, long) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845==  Address 0xffeffe050 is on thread 1's stack
==10845== 
==10845== Syscall param msync(start) points to unaddressable byte(s)
==10845==    at 0x4E46B3D: ??? (syscall-template.S:81)
==10845==    by 0x7C02123: ??? (in /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1)
==10845==    by 0x7C02B31: _ULx86_64_step (in /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1)
==10845==    by 0x5677442: google::GetStackTrace(void**, int, int) (in /usr/lib/x86_64-linux-gnu/libglog.so.0.0.0)
==10845==    by 0x567CB31: ??? (in /usr/lib/x86_64-linux-gnu/libglog.so.0.0.0)
==10845==    by 0x4E4733F: ??? (in /lib/x86_64-linux-gnu/libpthread-2.19.so)
==10845==    by 0x8B48FFF9A57AE80F: ???
==10845==    by 0x64F16E2: std::ostream::write(char const*, long) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845==    by 0x59089FB: Aws::Http::CurlHttpClient::WriteData(char*, unsigned long, unsigned long, void*) (in /usr/lib/libaws-cpp-sdk-core.so)
==10845==    by 0x7E2DCCF: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E474AF: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E41CD6: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==  Address 0x8b48fff9a57ae000 is not stack'd, malloc'd or (recently) free'd
==10845== 
==10845== Syscall param msync(start) points to unaddressable byte(s)
==10845==    at 0x4E46B3D: ??? (syscall-template.S:81)
==10845==    by 0x7C02123: ??? (in /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1)
==10845==    by 0x7C02BC1: _ULx86_64_step (in /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1)
==10845==    by 0x5677442: google::GetStackTrace(void**, int, int) (in /usr/lib/x86_64-linux-gnu/libglog.so.0.0.0)
==10845==    by 0x567CB31: ??? (in /usr/lib/x86_64-linux-gnu/libglog.so.0.0.0)
==10845==    by 0x4E4733F: ??? (in /lib/x86_64-linux-gnu/libpthread-2.19.so)
==10845==    by 0x8B48FFF9A57AE80F: ???
==10845==    by 0x64F16E2: std::ostream::write(char const*, long) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845==    by 0x59089FB: Aws::Http::CurlHttpClient::WriteData(char*, unsigned long, unsigned long, void*) (in /usr/lib/libaws-cpp-sdk-core.so)
==10845==    by 0x7E2DCCF: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E474AF: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E41CD6: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==  Address 0xe56f920 is 0 bytes after a block of size 35,984 alloc'd
==10845==    at 0x4C2CC70: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==10845==    by 0x7E34529: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x7E435F5: curl_easy_init (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845==    by 0x59083EF: Aws::Http::CurlHandleContainer::CheckAndGrowPool() (in /usr/lib/libaws-cpp-sdk-core.so)
==10845==    by 0x59085B1: Aws::Http::CurlHandleContainer::AcquireCurlHandle() (in /usr/lib/libaws-cpp-sdk-core.so)
==10845==    by 0x590964E: Aws::Http::CurlHttpClient::MakeRequest(Aws::Http::HttpRequest&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const (in /usr/lib/libaws-cpp-sdk-core.so)
==10845==    by 0x4354C3: NET_Basic_Test::TestBody() (test_net.cpp:45)
==10845==    by 0x46E252: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (gtest.cc:2090)
==10845==    by 0x466E66: testing::Test::Run() (gtest.cc:2162)
==10845==    by 0x466F0D: testing::TestInfo::Run() (gtest.cc:2338)
==10845==    by 0x467014: testing::TestCase::Run() (gtest.cc:2445)
==10845==    by 0x4672B7: testing::internal::UnitTestImpl::RunAllTests() (gtest.cc:4243)
==10845== 
*** SIGSEGV (@0x8b48fff9a57ae810) received by PID 10845 (TID 0x405f580) from PID 18446744072190879760; stack trace: ***
    @          0x4e47340 (unknown)
    @ 0x8b48fff9a57ae810 (unknown)
==10845== 
==10845== HEAP SUMMARY:
==10845==     in use at exit: 576,891 bytes in 6,082 blocks
==10845==   total heap usage: 8,543 allocs, 2,461 frees, 740,617 bytes allocated
==10845== 
==10845== LEAK SUMMARY:
==10845==    definitely lost: 0 bytes in 0 blocks
==10845==    indirectly lost: 0 bytes in 0 blocks
==10845==      possibly lost: 9,521 bytes in 211 blocks
==10845==    still reachable: 567,370 bytes in 5,871 blocks
==10845==         suppressed: 0 bytes in 0 blocks
==10845== Rerun with --leak-check=full to see details of leaked memory
==10845== 
==10845== For counts of detected and suppressed errors, rerun with: -v
==10845== Use --track-origins=yes to see where uninitialised values come from
==10845== ERROR SUMMARY: 7 errors from 7 contexts (suppressed: 0 from 0)

[JonathanHenson]

Not sure if this is the problem or not but I don't think ofstream is a
subclass of iostream
On Oct 1, 2015 1:13 PM, "Christopher J. Hanks" notifications@github.com
wrote:

So I tried to implement this, and it appears to be segfaulting.

#include <gtest/gtest.h>
#include <glog/logging.h>

#include
#include

#include "rcp_net/NetFile.hpp"

#include <aws/core/client/ClientConfiguration.h>
#include <aws/core/http/HttpClientFactory.h>
#include <aws/core/http/HttpClient.h>
#include <aws/core/http/HttpRequest.h>
#include <aws/core/http/HttpResponse.h>
using namespace Aws;using namespace Client;using namespace Http;

Aws::Client::ClientConfiguration Config;
TEST(NET, Basic) {
std::shared_ptr factory(new HttpClientFactory);
std::shared_ptr client(factory->CreateHttpClient(Config)); // sigh

std::string filePath("someTestPath");
size_t offset = 10;

std::shared_ptr<HttpRequest> request(
    factory->CreateHttpRequest(
        Aws::String("http://google.com"),
        HttpMethod::HTTP_GET,
        [&filePath, &offset]() -> Aws::IOStream* {
            constexpr auto OpenFlag = std::ios_base::out | std::ios_base::binary;
            Aws::OFStream* ofs = new Aws::OFStream(filePath, OpenFlag);
            CHECK(ofs->good());
            return dynamic_cast<Aws::IOStream*>(ofs);
        }));


std::shared_ptr<HttpResponse> response = client->MakeRequest(*request); //<-- *** Segfaults there ***
ASSERT_EQ(HttpResponseCode::OK,
          response->GetResponseCode());

$> valgrind --tool=memcheck ./gtests --gtest_filter='NET.'
==10845== Memcheck, a memory error detector
==10845== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==10845== Using Valgrind-3.10.0.SVN and LibVEX; rerun with -h for copyright info
==10845== Command: ./gtests --gtest_filter=NET.
==10845==
Note: Google Test filter = NET.*
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from NET
[ RUN ] NET.Basic
==10845== Invalid read of size 8
==10845== at 0x64F1309: std::ostream::sentry::sentry(std::ostream&) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845== by 0x64F16BD: std::ostream::write(char const_, long) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845== by 0x59089FB: Aws::Http::CurlHttpClient::WriteData(char_, unsigned long, unsigned long, void_) (in /usr/lib/libaws-cpp-sdk-core.so)
==10845== by 0x7E2DCCF: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845== by 0x7E474AF: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845== by 0x7E41CD6: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845== by 0x7E4B9DB: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845== by 0x7E4C180: curl_multi_perform (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845== by 0x7E437B2: curl_easy_perform (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845== by 0x59098E8: Aws::Http::CurlHttpClient::MakeRequest(Aws::Http::HttpRequest&, Aws::Utils::RateLimits::RateLimiterInterface_, Aws::Utils::RateLimits::RateLimiterInterface_) const (in /usr/lib/libaws-cpp-sdk-core.so)
==10845== by 0x4354C3: NET_Basic_Test::TestBody() (test_net.cpp:45)
==10845== by 0x46E252: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test_, void (testing::Test::)(), char const) (gtest.cc:2090)
==10845== Address 0xe570098 is 24 bytes before a block of size 8,192 alloc'd
==10845== at 0x4C2B800: operator new[](unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==10845== by 0x64D0497: std::basic_filebuf<char, std::char_traits >::M_allocate_internal_buffer() (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845== by 0x64D4541: std::basic_filebuf<char, std::char_traits >::open(char const, std::Ios_Openmode) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845== by 0x43511B: std::Function_handler<std::iostream (), NET_Basic_Test::TestBody()::{lambda()#1}>::M_invoke(std::Any_data const&) (fstream:281)
==10845== by 0x5911E40: Aws::Utils::Stream::ResponseStream::ResponseStream(std::function<std::iostream* ()> const&) (in /usr/lib/libaws-cpp-sdk-core.so)
==10845== by 0x5909745: Aws::Http::CurlHttpClient::MakeRequest(Aws::Http::HttpRequest&, Aws::Utils::RateLimits::RateLimiterInterface, Aws::Utils::RateLimits::RateLimiterInterface) const (in /usr/lib/libaws-cpp-sdk-core.so)
==10845== by 0x4354C3: NET_Basic_Test::TestBody() (test_net.cpp:45)
==10845== by 0x46E252: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test, void (testing::Test::)(), char const) (gtest.cc:2090)
==10845== by 0x466E66: testing::Test::Run() (gtest.cc:2162)
==10845== by 0x466F0D: testing::TestInfo::Run() (gtest.cc:2338)
==10845== by 0x467014: testing::TestCase::Run() (gtest.cc:2445)
==10845== by 0x4672B7: testing::internal::UnitTestImpl::RunAllTests() (gtest.cc:4243)
==10845==
==10845== Invalid read of size 8
==10845== at 0x64F132B: std::ostream::sentry::sentry(std::ostream&) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845== by 0x64F16BD: std::ostream::write(char const_, long) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845== by 0x59089FB: Aws::Http::CurlHttpClient::WriteData(char_, unsigned long, unsigned long, void_) (in /usr/lib/libaws-cpp-sdk-core.so)
==10845== by 0x7E2DCCF: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845== by 0x7E474AF: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845== by 0x7E41CD6: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845== by 0x7E4B9DB: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845== by 0x7E4C180: curl_multi_perform (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845== by 0x7E437B2: curl_easy_perform (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845== by 0x59098E8: Aws::Http::CurlHttpClient::MakeRequest(Aws::Http::HttpRequest&, Aws::Utils::RateLimits::RateLimiterInterface_, Aws::Utils::RateLimits::RateLimiterInterface_) const (in /usr/lib/libaws-cpp-sdk-core.so)
==10845== by 0x4354C3: NET_Basic_Test::TestBody() (test_net.cpp:45)
==10845== by 0x46E252: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test_, void (testing::Test::)(), char const) (gtest.cc:2090)
==10845== Address 0xe570098 is 24 bytes before a block of size 8,192 alloc'd
==10845== at 0x4C2B800: operator new[](unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==10845== by 0x64D0497: std::basic_filebuf<char, std::char_traits >::M_allocate_internal_buffer() (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845== by 0x64D4541: std::basic_filebuf<char, std::char_traits >::open(char const, std::Ios_Openmode) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845== by 0x43511B: std::Function_handler<std::iostream* (), NET_Basic_Test::TestBody()::{lambda()#1}>::M_invoke(std::Any_data const&) (fstream:281)
==10845== by 0x5911E40: Aws::Utils::Stream::ResponseStream::ResponseStream(std::function<std::iostream* ()> const&) (in /usr/lib/libaws-cpp-sdk-core.so)
==10845== by 0x5909745: Aws::Http::CurlHttpClient::MakeRequest(Aws::Http::HttpRequest&, Aws::Utils::RateLimits::RateLimiterInterface, Aws::Utils::RateLimits::RateLimiterInterface) const (in /usr/lib/libaws-cpp-sdk-core.so)
==10845== by 0x4354C3: NET_Basic_Test::TestBody() (test_net.cpp:45)
==10845== by 0x46E252: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test, void (testing::Test::)(), char const_) (gtest.cc:2090)
==10845== by 0x466E66: testing::Test::Run() (gtest.cc:2162)
==10845== by 0x466F0D: testing::TestInfo::Run() (gtest.cc:2338)
==10845== by 0x467014: testing::TestCase::Run() (gtest.cc:2445)
==10845== by 0x4672B7: testing::internal::UnitTestImpl::RunAllTests() (gtest.cc:4243)
==10845==
==10845== Invalid read of size 8
==10845== at 0x64F16D1: std::ostream::write(char const_, long) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845== by 0x59089FB: Aws::Http::CurlHttpClient::WriteData(char_, unsigned long, unsigned long, void_) (in /usr/lib/libaws-cpp-sdk-core.so)
==10845== by 0x7E2DCCF: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845== by 0x7E474AF: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845== by 0x7E41CD6: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845== by 0x7E4B9DB: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845== by 0x7E4C180: curl_multi_perform (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845== by 0x7E437B2: curl_easy_perform (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845== by 0x59098E8: Aws::Http::CurlHttpClient::MakeRequest(Aws::Http::HttpRequest&, Aws::Utils::RateLimits::RateLimiterInterface_, Aws::Utils::RateLimits::RateLimiterInterface_) const (in /usr/lib/libaws-cpp-sdk-core.so)
==10845== by 0x4354C3: NET_Basic_Test::TestBody() (test_net.cpp:45)
==10845== by 0x46E252: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test_, void (testing::Test::)(), char const) (gtest.cc:2090)
==10845== by 0x466E66: testing::Test::Run() (gtest.cc:2162)
==10845== Address 0xe570098 is 24 bytes before a block of size 8,192 alloc'd
==10845== at 0x4C2B800: operator new[](unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==10845== by 0x64D0497: std::basic_filebuf<char, std::char_traits >::M_allocate_internal_buffer() (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845== by 0x64D4541: std::basic_filebuf<char, std::char_traits >::open(char const, std::Ios_Openmode) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845== by 0x43511B: std::Function_handler<std::iostream* (), NET_Basic_Test::TestBody()::{lambda()#1}>::M_invoke(std::Any_data const&) (fstream:281)
==10845== by 0x5911E40: Aws::Utils::Stream::ResponseStream::ResponseStream(std::function<std::iostream* ()> const&) (in /usr/lib/libaws-cpp-sdk-core.so)
==10845== by 0x5909745: Aws::Http::CurlHttpClient::MakeRequest(Aws::Http::HttpRequest&, Aws::Utils::RateLimits::RateLimiterInterface, Aws::Utils::RateLimits::RateLimiterInterface) const (in /usr/lib/libaws-cpp-sdk-core.so)
==10845== by 0x4354C3: NET_Basic_Test::TestBody() (test_net.cpp:45)
==10845== by 0x46E252: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test, void (testing::Test::)(), char const_) (gtest.cc:2090)
==10845== by 0x466E66: testing::Test::Run() (gtest.cc:2162)
==10845== by 0x466F0D: testing::TestInfo::Run() (gtest.cc:2338)
==10845== by 0x467014: testing::TestCase::Run() (gtest.cc:2445)
==10845== by 0x4672B7: testing::internal::UnitTestImpl::RunAllTests() (gtest.cc:4243)
==10845==
==10845== Jump to the invalid address stated on the next line
==10845== at 0x8B48FFF9A57AE810: ???
==10845== by 0x64F16E2: std::ostream::write(char const_, long) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845== by 0x59089FB: Aws::Http::CurlHttpClient::WriteData(char_, unsigned long, unsigned long, void_) (in /usr/lib/libaws-cpp-sdk-core.so)
==10845== by 0x7E2DCCF: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845== by 0x7E474AF: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845== by 0x7E41CD6: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845== by 0x7E4B9DB: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845== by 0x7E4C180: curl_multi_perform (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845== by 0x7E437B2: curl_easy_perform (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845== by 0x59098E8: Aws::Http::CurlHttpClient::MakeRequest(Aws::Http::HttpRequest&, Aws::Utils::RateLimits::RateLimiterInterface_, Aws::Utils::RateLimits::RateLimiterInterface_) const (in /usr/lib/libaws-cpp-sdk-core.so)
==10845== by 0x4354C3: NET_Basic_Test::TestBody() (test_net.cpp:45)
==10845== by 0x46E252: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test_, void (testing::Test::)(), char const) (gtest.cc:2090)
==10845== Address 0x8b48fff9a57ae810 is not stack'd, malloc'd or (recently) free'd
==10845==
_* Aborted at 1443730215 (unix time) try "date -d @1443730215" if you are using GNU date ***
PC: @ 0x8b48fff9a57ae810 (unknown)
==10845== Syscall param msync(start) points to uninitialised byte(s)
==10845== at 0x4E46B3D: ??? (syscall-template.S:81)
==10845== by 0x7C02123: ??? (in /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1)
==10845== by 0x7C02643: ??? (in /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1)
==10845== by 0x7C04EAA: ??? (in /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1)
==10845== by 0x7C06151: ??? (in /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1)
==10845== by 0x7C064E8: ??? (in /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1)
==10845== by 0x7C02A30: _ULx86_64_step (in /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1)
==10845== by 0x5677442: google::GetStackTrace(void*, int, int) (in /usr/lib/x86_64-linux-gnu/libglog.so.0.0.0)
==10845== by 0x567CB31: ??? (in /usr/lib/x86_64-linux-gnu/libglog.so.0.0.0)
==10845== by 0x4E4733F: ??? (in /lib/x86_64-linux-gnu/libpthread-2.19.so)
==10845== by 0x8B48FFF9A57AE80F: ???
==10845== by 0x64F16E2: std::ostream::write(char const*, long) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845== Address 0xffeffe050 is on thread 1's stack
==10845==
==10845== Syscall param msync(start) points to unaddressable byte(s)
==10845== at 0x4E46B3D: ??? (syscall-template.S:81)
==10845== by 0x7C02123: ??? (in /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1)
==10845== by 0x7C02B31: ULx86_64_step (in /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1)
==10845== by 0x5677442: google::GetStackTrace(void**, int, int) (in /usr/lib/x86_64-linux-gnu/libglog.so.0.0.0)
==10845== by 0x567CB31: ??? (in /usr/lib/x86_64-linux-gnu/libglog.so.0.0.0)
==10845== by 0x4E4733F: ??? (in /lib/x86_64-linux-gnu/libpthread-2.19.so)
==10845== by 0x8B48FFF9A57AE80F: ???
==10845== by 0x64F16E2: std::ostream::write(char const, long) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845== by 0x59089FB: Aws::Http::CurlHttpClient::WriteData(char_, unsigned long, unsigned long, void_) (in /usr/lib/libaws-cpp-sdk-core.so)
==10845== by 0x7E2DCCF: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845== by 0x7E474AF: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845== by 0x7E41CD6: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845== Address 0x8b48fff9a57ae000 is not stack'd, malloc'd or (recently) free'd
==10845==
==10845== Syscall param msync(start) points to unaddressable byte(s)
==10845== at 0x4E46B3D: ??? (syscall-template.S:81)
==10845== by 0x7C02123: ??? (in /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1)
==10845== by 0x7C02BC1: ULx86_64_step (in /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1)
==10845== by 0x5677442: google::GetStackTrace(void_, int, int) (in /usr/lib/x86_64-linux-gnu/libglog.so.0.0.0)
==10845== by 0x567CB31: ??? (in /usr/lib/x86_64-linux-gnu/libglog.so.0.0.0)
==10845== by 0x4E4733F: ??? (in /lib/x86_64-linux-gnu/libpthread-2.19.so)
==10845== by 0x8B48FFF9A57AE80F: ???
==10845== by 0x64F16E2: std::ostream::write(char const_, long) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==10845== by 0x59089FB: Aws::Http::CurlHttpClient::WriteData(char_, unsigned long, unsigned long, void_) (in /usr/lib/libaws-cpp-sdk-core.so)
==10845== by 0x7E2DCCF: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845== by 0x7E474AF: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845== by 0x7E41CD6: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845== Address 0xe56f920 is 0 bytes after a block of size 35,984 alloc'd
==10845== at 0x4C2CC70: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==10845== by 0x7E34529: ??? (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845== by 0x7E435F5: curl_easy_init (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==10845== by 0x59083EF: Aws::Http::CurlHandleContainer::CheckAndGrowPool() (in /usr/lib/libaws-cpp-sdk-core.so)
==10845== by 0x59085B1: Aws::Http::CurlHandleContainer::AcquireCurlHandle() (in /usr/lib/libaws-cpp-sdk-core.so)
==10845== by 0x590964E: Aws::Http::CurlHttpClient::MakeRequest(Aws::Http::HttpRequest&, Aws::Utils::RateLimits::RateLimiterInterface_, Aws::Utils::RateLimits::RateLimiterInterface_) const (in /usr/lib/libaws-cpp-sdk-core.so)
==10845== by 0x4354C3: NET_Basic_Test::TestBody() (test_net.cpp:45)
==10845== by 0x46E252: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test_, void (testing::Test::)(), char const) (gtest.cc:2090)
==10845== by 0x466E66: testing::Test::Run() (gtest.cc:2162)
==10845== by 0x466F0D: testing::TestInfo::Run() (gtest.cc:2338)
==10845== by 0x467014: testing::TestCase::Run() (gtest.cc:2445)
==10845== by 0x4672B7: testing::internal::UnitTestImpl::RunAllTests() (gtest.cc:4243)
==10845==
*_* SIGSEGV (@0x8b48fff9a57ae810) received by PID 10845 (TID 0x405f580) from PID 18446744072190879760; stack trace: ***
@ 0x4e47340 (unknown)
@ 0x8b48fff9a57ae810 (unknown)
==10845==
==10845== HEAP SUMMARY:
==10845== in use at exit: 576,891 bytes in 6,082 blocks
==10845== total heap usage: 8,543 allocs, 2,461 frees, 740,617 bytes allocated
==10845==
==10845== LEAK SUMMARY:
==10845== definitely lost: 0 bytes in 0 blocks
==10845== indirectly lost: 0 bytes in 0 blocks
==10845== possibly lost: 9,521 bytes in 211 blocks
==10845== still reachable: 567,370 bytes in 5,871 blocks
==10845== suppressed: 0 bytes in 0 blocks
==10845== Rerun with --leak-check=full to see details of leaked memory
==10845==
==10845== For counts of detected and suppressed errors, rerun with: -v
==10845== Use --track-origins=yes to see where uninitialised values come from
==10845== ERROR SUMMARY: 7 errors from 7 contexts (suppressed: 0 from 0)

—
Reply to this email directly or view it on GitHub
#21 (comment).

[cjhanks]

Yes apologies, I have corrected the example so that no wild casting is required (that was a clip from one of my debugging permutations).

[JonathanHenson]

Did you rerun after making that change?
On Oct 1, 2015 1:19 PM, "Christopher J. Hanks" notifications@github.com
wrote:

Yes apologies, I have corrected the example so that no wild casting is
required (that was a clip from one of my debugging permutations).

—
Reply to this email directly or view it on GitHub
#21 (comment).

[cjhanks]

==11261== Memcheck, a memory error detector
==11261== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==11261== Using Valgrind-3.10.0.SVN and LibVEX; rerun with -h for copyright info
==11261== Command: ./gtests --gtest_filter=NET.*
==11261== 
Note: Google Test filter = NET.*
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from NET
[ RUN      ] NET.Basic
==11261== Mismatched free() / delete / delete []
==11261==    at 0x4C2BDEC: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==11261==    by 0x5911E80: Aws::Utils::Stream::ResponseStream::ReleaseStream() (in /usr/lib/libaws-cpp-sdk-core.so)
==11261==    by 0x5907C86: Aws::Http::Standard::StandardHttpResponse::~StandardHttpResponse() (in /usr/lib/libaws-cpp-sdk-core.so)
==11261==    by 0x435B78: NET_Basic_Test::TestBody() (shared_ptr_base.h:144)
==11261==    by 0x46E602: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (gtest.cc:2090)
==11261==    by 0x467216: testing::Test::Run() (gtest.cc:2162)
==11261==    by 0x4672BD: testing::TestInfo::Run() (gtest.cc:2338)
==11261==    by 0x4673C4: testing::TestCase::Run() (gtest.cc:2445)
==11261==    by 0x467667: testing::internal::UnitTestImpl::RunAllTests() (gtest.cc:4243)
==11261==    by 0x467906: testing::UnitTest::Run() (gtest.cc:2090)
==11261==    by 0x410EA1: main (main.cpp:12)
==11261==  Address 0xe56fbf0 is 0 bytes inside a block of size 528 alloc'd
==11261==    at 0x4C2B0E0: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==11261==    by 0x4350CA: std::_Function_handler<std::iostream* (), NET_Basic_Test::TestBody()::{lambda()#1}>::_M_invoke(std::_Any_data const&) (test_net.cpp:36)
==11261==    by 0x5911E40: Aws::Utils::Stream::ResponseStream::ResponseStream(std::function<std::iostream* ()> const&) (in /usr/lib/libaws-cpp-sdk-core.so)
==11261==    by 0x5909745: Aws::Http::CurlHttpClient::MakeRequest(Aws::Http::HttpRequest&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const (in /usr/lib/libaws-cpp-sdk-core.so)
==11261==    by 0x4355C4: NET_Basic_Test::TestBody() (test_net.cpp:41)
==11261==    by 0x46E602: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (gtest.cc:2090)
==11261==    by 0x467216: testing::Test::Run() (gtest.cc:2162)
==11261==    by 0x4672BD: testing::TestInfo::Run() (gtest.cc:2338)
==11261==    by 0x4673C4: testing::TestCase::Run() (gtest.cc:2445)
==11261==    by 0x467667: testing::internal::UnitTestImpl::RunAllTests() (gtest.cc:4243)
==11261==    by 0x467906: testing::UnitTest::Run() (gtest.cc:2090)
==11261==    by 0x410EA1: main (main.cpp:12)
==11261== 
==11261== Invalid free() / delete / delete[] / realloc()
==11261==    at 0x4C2C2BC: operator delete(void*) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==11261==    by 0x435D0E: NET_Basic_Test::TestBody() (basic_string.h:249)
==11261==    by 0x46E602: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (gtest.cc:2090)
==11261==    by 0x467216: testing::Test::Run() (gtest.cc:2162)
==11261==    by 0x4672BD: testing::TestInfo::Run() (gtest.cc:2338)
==11261==    by 0x4673C4: testing::TestCase::Run() (gtest.cc:2445)
==11261==    by 0x467667: testing::internal::UnitTestImpl::RunAllTests() (gtest.cc:4243)
==11261==    by 0x467906: testing::UnitTest::Run() (gtest.cc:2090)
==11261==    by 0x410EA1: main (main.cpp:12)
==11261==  Address 0x5b30500 is in the BSS segment of /usr/lib/libaws-cpp-sdk-core.so
==11261== 
==11261== Invalid free() / delete / delete[] / realloc()
==11261==    at 0x4C2C2BC: operator delete(void*) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==11261==    by 0x435DCE: NET_Basic_Test::TestBody() (basic_string.h:249)
==11261==    by 0x46E602: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (gtest.cc:2090)
==11261==    by 0x467216: testing::Test::Run() (gtest.cc:2162)
==11261==    by 0x4672BD: testing::TestInfo::Run() (gtest.cc:2338)
==11261==    by 0x4673C4: testing::TestCase::Run() (gtest.cc:2445)
==11261==    by 0x467667: testing::internal::UnitTestImpl::RunAllTests() (gtest.cc:4243)
==11261==    by 0x467906: testing::UnitTest::Run() (gtest.cc:2090)
==11261==    by 0x410EA1: main (main.cpp:12)
==11261==  Address 0x5b30500 is in the BSS segment of /usr/lib/libaws-cpp-sdk-core.so
==11261== 
==11261== Invalid free() / delete / delete[] / realloc()
==11261==    at 0x4C2C2BC: operator delete(void*) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==11261==    by 0x435DF7: NET_Basic_Test::TestBody() (basic_string.h:249)
==11261==    by 0x46E602: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (gtest.cc:2090)
==11261==    by 0x467216: testing::Test::Run() (gtest.cc:2162)
==11261==    by 0x4672BD: testing::TestInfo::Run() (gtest.cc:2338)
==11261==    by 0x4673C4: testing::TestCase::Run() (gtest.cc:2445)
==11261==    by 0x467667: testing::internal::UnitTestImpl::RunAllTests() (gtest.cc:4243)
==11261==    by 0x467906: testing::UnitTest::Run() (gtest.cc:2090)
==11261==    by 0x410EA1: main (main.cpp:12)
==11261==  Address 0x5b30500 is in the BSS segment of /usr/lib/libaws-cpp-sdk-core.so
==11261== 
==11261== Invalid free() / delete / delete[] / realloc()
==11261==    at 0x4C2C2BC: operator delete(void*) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==11261==    by 0x435DA5: NET_Basic_Test::TestBody() (basic_string.h:249)
==11261==    by 0x46E602: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (gtest.cc:2090)
==11261==    by 0x467216: testing::Test::Run() (gtest.cc:2162)
==11261==    by 0x4672BD: testing::TestInfo::Run() (gtest.cc:2338)
==11261==    by 0x4673C4: testing::TestCase::Run() (gtest.cc:2445)
==11261==    by 0x467667: testing::internal::UnitTestImpl::RunAllTests() (gtest.cc:4243)
==11261==    by 0x467906: testing::UnitTest::Run() (gtest.cc:2090)
==11261==    by 0x410EA1: main (main.cpp:12)
==11261==  Address 0x5b30500 is in the BSS segment of /usr/lib/libaws-cpp-sdk-core.so
==11261== 
==11261== Mismatched free() / delete / delete []
==11261==    at 0x4C2C2BC: operator delete(void*) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==11261==    by 0x435E20: NET_Basic_Test::TestBody() (basic_string.h:249)
==11261==    by 0x46E602: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (gtest.cc:2090)
==11261==    by 0x467216: testing::Test::Run() (gtest.cc:2162)
==11261==    by 0x4672BD: testing::TestInfo::Run() (gtest.cc:2338)
==11261==    by 0x4673C4: testing::TestCase::Run() (gtest.cc:2445)
==11261==    by 0x467667: testing::internal::UnitTestImpl::RunAllTests() (gtest.cc:4243)
==11261==    by 0x467906: testing::UnitTest::Run() (gtest.cc:2090)
==11261==    by 0x410EA1: main (main.cpp:12)
==11261==  Address 0xe5089f0 is 0 bytes inside a block of size 84 alloc'd
==11261==    at 0x4C2AB80: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==11261==    by 0x58E5D87: char* std::basic_string<char, std::char_traits<char>, Aws::Allocator<char> >::_S_construct<char*>(char*, char*, Aws::Allocator<char> const&, std::forward_iterator_tag) (in /usr/lib/libaws-cpp-sdk-core.so)
==11261==    by 0x58FD22B: Aws::Client::ComputeUserAgentString() (in /usr/lib/libaws-cpp-sdk-core.so)
==11261==    by 0x58FD50D: Aws::Client::ClientConfiguration::ClientConfiguration() (in /usr/lib/libaws-cpp-sdk-core.so)
==11261==    by 0x43545B: NET_Basic_Test::TestBody() (test_net.cpp:23)
==11261==    by 0x46E602: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (gtest.cc:2090)
==11261==    by 0x467216: testing::Test::Run() (gtest.cc:2162)
==11261==    by 0x4672BD: testing::TestInfo::Run() (gtest.cc:2338)
==11261==    by 0x4673C4: testing::TestCase::Run() (gtest.cc:2445)
==11261==    by 0x467667: testing::internal::UnitTestImpl::RunAllTests() (gtest.cc:4243)
==11261==    by 0x467906: testing::UnitTest::Run() (gtest.cc:2090)
==11261==    by 0x410EA1: main (main.cpp:12)
==11261== 
[       OK ] NET.Basic (1543 ms)
[----------] 1 test from NET (1558 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test case ran. (1587 ms total)
[  PASSED  ] 1 test.
==11261== 
==11261== HEAP SUMMARY:
==11261==     in use at exit: 180,644 bytes in 3,357 blocks
==11261==   total heap usage: 8,593 allocs, 5,240 frees, 744,842 bytes allocated
==11261== 
==11261== LEAK SUMMARY:
==11261==    definitely lost: 0 bytes in 0 blocks
==11261==    indirectly lost: 0 bytes in 0 blocks
==11261==      possibly lost: 1,652 bytes in 25 blocks
==11261==    still reachable: 178,992 bytes in 3,332 blocks
==11261==         suppressed: 0 bytes in 0 blocks
==11261== Rerun with --leak-check=full to see details of leaked memory
==11261== 
==11261== For counts of detected and suppressed errors, rerun with: -v
==11261== ERROR SUMMARY: 6 errors from 6 contexts (suppressed: 0 from 0)

So it is "PASS"ing with valgrind, but definitely failing.

*** Error in `./gtests': free(): invalid pointer: 0x00007f38bf3e5500 ***
*** Aborted at 1443730984 (unix time) try "date -d @1443730984" if you are using GNU date ***
PC: @     0x7f38bdc6dcc9 (unknown)
*** SIGABRT (@0x3e800002c1a) received by PID 11290 (TID 0x7f38c0028780) from PID 11290; stack trace: ***
    @     0x7f38bfc37340 (unknown)
    @     0x7f38bdc6dcc9 (unknown)
    @     0x7f38bdc710d8 (unknown)
    @     0x7f38bdcaa394 (unknown)
    @     0x7f38bdcb666e (unknown)
    @           0x435d0f NET_Basic_Test::TestBody()

    @           0x46e603 testing::internal::HandleExceptionsInMethodIfSupported<>()
    @           0x467217 testing::Test::Run()
    @           0x4672be testing::TestInfo::Run()
    @           0x4673c5 testing::TestCase::Run()
    @           0x467668 testing::internal::UnitTestImpl::RunAllTests()
    @           0x467907 testing::UnitTest::Run()
    @           0x410ea2 main
    @     0x7f38bdc58ec5 (unknown)
    @           0x413d07 (unknown)
    @                0x0 (unknown)

It appears that the error is occurring during some form of static destructor from within the AWS library, because my request completes successfully at this point.