Coding: Fleeting Thoughts

A place to discuss the implementation and style of computer programs.

Moderators: phlip, Moderators General, Prelates

Tub
Posts: 410
Joined: Wed Jul 27, 2011 3:13 pm UTC

Re: Coding: Fleeting Thoughts

Postby Tub » Tue Nov 20, 2018 5:53 am UTC

There are a bunch of nice css transforms, powerful enough to create a full animated 3d layout, if you like. When clicking somewhere, the browser will reverse those transforms, accurately determining which element was under the mouse pointer at the time of the event.
At the same time, the browser will also absolutely refuse to give you any helpful coordinates telling you *where* the element has been clicked, so you either have to reimplement the transformation matrix formulas and do it yourself, or you have to overlay your widgets in a grid of 1x1 divs and hope that your clients don't notice any slowdown. *sigh*

User avatar
Xenomortis
Not actually a special flower.
Posts: 1426
Joined: Thu Oct 11, 2012 8:47 am UTC

Re: Coding: Fleeting Thoughts

Postby Xenomortis » Tue Nov 20, 2018 11:46 pm UTC

My knowledge of the details of dynamic linking is limited; I don't really know what happens when "dlopen" is called.
But the following situation occurs with some program at work, and my intuition tells me that "bad things" could happen.

There exists an application (call it "app"). It links to "foo_v1.so"

Code: Select all

$ ldd app
...
foo_v1.so => [path/to/foo_v1.so] [hex address]
...

foo_v1.so may, at some point during the execution of app, open another shared object "bar.so".
bar.so however, links to foo_v2.so - a different version of foo.

Code: Select all

$ ldd foo_v1.so
...
foo_v1.so
...


Library foo holds a fair amount of internal state, some of which I believe is allocated statically. Calls to functions in foo will often modify this internal state. foo is not threadsafe (by default anyway).

So, what's happening here?
Are there two instances of "foo" loaded? What happens with the symbol collisions? What gets used when bar wants a function from foo - which version does it get?
Why doesn't "app" occasionally explode? Why do there appear to be no consequences?
Are we just lucky that bar doesn't actually call much from foo?

Edit:
Functions are resolved as offsets from the load point of the library? So bar only "sees" foo_v2?
Image

Tub
Posts: 410
Joined: Wed Jul 27, 2011 3:13 pm UTC

Re: Coding: Fleeting Thoughts

Postby Tub » Wed Nov 21, 2018 6:34 pm UTC

Xenomortis wrote:my intuition tells me that "bad things" could happen.

That is a reasonable default assumption. If you wish to dive into the inner workings of symbol resolution, read this: https://akkadia.org/drepper/dsohowto.pdf

>> Are there two instances of "foo" loaded?

Yes. Unless they're the same file, they're a different library. The fact that their names both contain "foo" doesn't mean anything.

>> What happens with the symbol collisions?

If v1 and v2 have a different API/ABI, then those symbols *should* be versioned, and you should only link against versioned symbols, hence there would be no symbol collision.

Otherwise, symbol resolution simply returns the first match, but with a well defined search order.

Generelly, the linker just returns void pointers. The linker does not guarantee that the symbol you're getting complies with the ABI that you're expecting. It's the system maintainer's responsibility to ensure that the exported symbols in the search path match the expected symbols of the installed executables.

>> What gets used when bar wants a function from foo - which version does it get?

Page 10 of the pdf linked above deals with that specific case. The global scope is searched before the local scope introduced during dlopen() - if your symbols are unversioned, it gets the version that your app was linked against, not the one that bar links against.

>> Why doesn't "app" occasionally explode? Why do there appear to be no consequences?

Either because those symbols are versioned, or because you've been lucky, or because you've been unlucky and those bugs will only turn up in production while you're on vacation.

Library foo holds a fair amount of internal state, some of which I believe is allocated statically.

If both your app and bar.so link against foo.so, which has a global state, then you can get data races on that global state without even introducing a thread. In addition to being thread-safe, you also need foo to be reentrant.

bar.so linking against a different version (thus getting its own internal state) may work in your favour here, but that's obviously not a good thing to rely on.

User avatar
Yakk
Poster with most posts but no title.
Posts: 11092
Joined: Sat Jan 27, 2007 7:27 pm UTC
Location: E pur si muove

Re: Coding: Fleeting Thoughts

Postby Yakk » Wed Nov 21, 2018 7:28 pm UTC

So, in C++ land you can safely version your libraries like this:

Code: Select all

namespace my_library_ns {
  inline namespace version2_0_1 {
    // API goes here
  }
}

now clients can just use `my_library_ns::foo()` and not even know about the `inline` namespace. But they link against `my_library_ns::version2_0_1::foo` magically.
One of the painful things about our time is that those who feel certainty are stupid, and those with any imagination and understanding are filled with doubt and indecision - BR

Last edited by JHVH on Fri Oct 23, 4004 BCE 6:17 pm, edited 6 times in total.

User avatar
Xenomortis
Not actually a special flower.
Posts: 1426
Joined: Thu Oct 11, 2012 8:47 am UTC

Re: Coding: Fleeting Thoughts

Postby Xenomortis » Wed Nov 21, 2018 7:55 pm UTC

Yakk wrote:So, in C++...

"foo" and "bar" are C (and we can compile both). However "app" appears to be C++ (we don't have the secret sauce) and was distributed with its own version of "foo".


Tub wrote:>> What happens with the symbol collisions?

If v1 and v2 have a different API/ABI, then those symbols *should* be versioned, and you should only link against versioned symbols, hence there would be no symbol collision.

Otherwise, symbol resolution simply returns the first match, but with a well defined search order.

Generelly, the linker just returns void pointers. The linker does not guarantee that the symbol you're getting complies with the ABI that you're expecting. It's the system maintainer's responsibility to ensure that the exported symbols in the search path match the expected symbols of the installed executables.

The ABI's consistent - I guess the question is, what objects get modified by the different libraries. What happens with their internal data?

Suppose then (no longer completely representative of the case at hand):
foo_v1.c:

Code: Select all

static int x = 0;
void foo_f1() {
    x++;
}

and foo_v2.c:

Code: Select all

static int x = 0;
void foo_f1() {
    return ++x;
}
void foo_f2() {
    x *= -1;
    return x;
}


bar.c

Code: Select all

int bar_f() {
  foo_f1();
  return foo_f2();
}


app.c

Code: Select all

int main() {
  foo_f1();
  /*
  ..dlopen(bar.so) etc..
  */
  int y = bar_f();
}


What's the value of y in app.c? I suppose "-2", but that assumes foo_f2 read the same x that foo_f1 (that would have come from the global scope).
And what would it be if foo_v2.c looked like:

Code: Select all

static int x = 0;
static int y = 2;
void foo_f1() {
    return ++x;
}
void foo_f2() {
    x *= -1;
    return x + y;
}


Tub wrote:If you wish to dive into the inner workings of symbol resolution, read this: https://akkadia.org/drepper/dsohowto.pdf

I'll try to, but maybe not after beer. ;)
Image

Tub
Posts: 410
Joined: Wed Jul 27, 2011 3:13 pm UTC

Re: Coding: Fleeting Thoughts

Postby Tub » Wed Nov 21, 2018 8:53 pm UTC

Xenomortis wrote:The ABI's consistent

If that is the case, then you'd expect v1 to be a symlink to v2, and you wouldn't have this trouble. If you wish to override a vendor-supplied binary, you can use LD_PRELOAD to make app use your system's v2.

Xenomortis wrote:"foo" and "bar" are C (and we can compile both). However "app" appears to be C++ (we don't have the secret sauce) and was distributed with its own version of "foo".

If you have the source of foo, then you should know if it has global internal state, if it's thread-safe/reentrant, and how it solves its own versioning.

You can force bar to use a separate copy of foo by statically linking, if that avoids trouble.

>> Suppose then (no longer completely representative of the case at hand):

assuming that code compiles (it doesn't):

* if the symbols are versioned, bar gets both functions from f2, and y will be -1
* if the symbols are unversioned, bar would be linked with foo_f1 from v1, and foo_f2 from v2, and y is 0.

There is no case where y = -2. x cannot be shared between both versions unless it's exported.

>> And what would it be if foo_v2.c looked like:

results are two larger than above.

User avatar
Xeio
Friends, Faidites, Countrymen
Posts: 5099
Joined: Wed Jul 25, 2007 11:12 am UTC
Location: C:\Users\Xeio\
Contact:

Re: Coding: Fleeting Thoughts

Postby Xeio » Thu Nov 29, 2018 6:45 pm UTC

Sometimes when I see reflection use I think "Has science gone too far?".

Like I mean it's cool to get every type from the current assembly and check if they have a particular static method then invoking the method... I guess...

But there's only one of those aforementioned types in the whole solution so I mean, they could have just had a line to call it directly. I guess in the future if/when we add more you're saved one line of code to the places these are all called from... but... Ehhhh.

EDIT: Apparently this is for an implementation of IRegisterMetadata which already works like the above, so there's a reflection loop looking for implementations of IRegisterMetadata which in itself is looking for implementations of this particular static method. I'm not sure why the XAML desigenrs wouldn't just implement IRegisterMetadata directly...

I'll just choose to believe that there is some unknown performance reason to throw away type safety and add fragility that if someone ever creates an unrelated static method with the same name in the assembly that the whole thing blows up.

User avatar
Link
Posts: 1366
Joined: Sat Mar 07, 2009 11:33 am UTC
Location: ᘝᓄᘈᖉᐣ
Contact:

Re: Coding: Fleeting Thoughts

Postby Link » Fri Nov 30, 2018 11:34 am UTC

Gahh, kludges, kludges everywhere. Eigen's matrices can use custom complex types, but ComplexEigenSolver requires std::complex (possibly of a custom real type), so now I have to use kludges to convert my mpc-based complex numbers (via boost::multiprecision::mpc_complex) to std::complex of an mpfr-derived real type to compute the eigenvalues and -vectors, and then convert those back to mpc. Looks like there's a bug report for this, but in the mean time I have use this stupid kludge. Bollocks.

EDIT: completely unrelated, but in modern C++, what is the consensus on defining inline functions in header files: good practise or bad? With template bodies already needing to be put in header files anyway, and inline being only a hint to the compiler, it's tempting to just slap an inline keyword on every function that needs to be used in multiple source files and put the whole definition in the header (this is a well-contained project that doesn't need to expose an API or provide a library to other projects). Since modern compilers seem to be smart enough to figure out function call optimisation on their own most of the time, I don't see any significant downsides to doing so, but I still have the old "declaration in the header, definition in the source" mantra firmly rooted in the back of my mind, so it's making me a bit uncomfortable to break it so heavily.

Tub
Posts: 410
Joined: Wed Jul 27, 2011 3:13 pm UTC

Re: Coding: Fleeting Thoughts

Postby Tub » Sat Dec 01, 2018 12:29 pm UTC

Link wrote:EDIT: completely unrelated, but in modern C++, what is the consensus on defining inline functions in header files: good practise or bad?

.cpp file: "Eh, I guess a really smart compiler could still inline it"
.h file: as inline "I trust the compiler to make a good choice. The linker will remove the redundant non-inline instantiations, right?"
.h file, as static inline: "That should save the linker some trouble."
.h file, #define: "I want to make it as difficult as possible for the compiler to disobey my wishes."

It depends on your toolchain's capabilities for link-time optimization, your compiler's ability to make good inlining decisions (see: pgo), and your personal priorities regarding compilation speed vs. running speed vs. binary size vs. code style. If in doubt, compare both using your favourite toolchain.

User avatar
Flumble
Yes Man
Posts: 2082
Joined: Sun Aug 05, 2012 9:35 pm UTC

Re: Coding: Fleeting Thoughts

Postby Flumble » Tue Dec 04, 2018 3:12 am UTC

Not really coding, but I don't see a "helpdesk: fleeting thoughts":
I'm copying (cp) some large files from a remote computer (over sshfs) to an encrypted partition (luks/dmcrypt stuff) and it keeps switching between download & 'write', and encrypt & write.
During the former ssh, sshfs and cp are quite busy: top reports cpu use up to 50%, nethogs says 100MB/s traffic, and iotop says both total disk read and write are 100MB/s (and actual reads and writes at 0).
During the latter some kcryptds and dmcrypt_write get busy: 10% cpu use per kcryptd, no network traffic, and iotop reports some incredible peaks in actual writes (up to 2GB/s) while staying at 0 for most of the time and for the other statistics.

Why is the computer being so silly? And who is to blame/can it be remedied or worked around?
I've tried mounting the encrypted partition with sync, but, while it seemingly removed this switching behaviour, the throughput was an abysmal 5MB/s.


(Note: both disks are old-fashioned spinny things with read and write speeds of about 100MB/s.)
(Note: I disabled the write cache on the disk for reasons; dunno if that affects the situation.)

Tub
Posts: 410
Joined: Wed Jul 27, 2011 3:13 pm UTC

Re: Coding: Fleeting Thoughts

Postby Tub » Tue Dec 04, 2018 12:47 pm UTC

This could be perfectly harmless, if your total average throughput matches what the disk supports. In a perfect world, network speed would adjust to disk speed, but several layers of buffers and queues may prevent that.
IIRC there are a few procfs/sysctl knobs you can try to influence dirty page handling. Disabling the write cache may have influenced the behaviour, too.
But the first thing I'd try is to get rid of buffers in sshfs/fuse. How does scp and/or rsync work for you?


Return to “Coding”

Who is online

Users browsing this forum: No registered users and 12 guests