Coding: Fleeting Thoughts

[fizzgig]

I think you hit it.

IosContext is giving out its internal Calendar instance - an easy to make mistake for novice to intermediate programmers.
When it was noticed that this means anybody can set the program-wide date, it got fixed by tellling everybody they have to copy the returned Calendar before using it.

I honestly doubt that is actually the case. IosContext would be calling a web service. You might be able to modify the Calendar object you get back, but it's not going to change the actual system date.

Is it giving out a new instance on every call?
And has it done so in all earlier versions?

These are questions to which I have no answer I'm afraid. Especially on a Saturday afternoon.

[Markus__1]

...

These are questions to which I have no answer I'm afraid. Especially on a Saturday afternoon.

No problem, these are idle speculations anyway.

I just think I have got a feeling for how such strangeness comes about from reading (too much ) DailyWTF - especially comments that explain it with some similar project's history.

[TheChewanater]

FT: I just upgraded to the latest Ubuntu the other day, which upgraded me to gcc 4.6.1. Now all of a sudden I get a linker error for every function from an external library, unless I put the "-lLibrary" flags AFTER the source files on the command line. This wasted a whole day of potential coding time. WTF, gcc?

[Tegelane]

GCC is not to blame here. The linker is doing this is to keep your binaries smaller. It only links in libraries that satisfy some already encountered unresolved function. So if you put the libs first on the command line the linker sees, that nothing actually needs it yet and drops it completely. With static libraries it gives a huge saving, since the linker can pick out only the individual needed object files.

[headprogrammingczar]

You are missing the point, which is "why should the order of parameters have any effect on what GCC does?".

[Tegelane]

And you're missing my point GCC does nothing with library files - it just passes them along to the linker. And gcc and binutils are two separate projects.

[Steax]

More sanity-checking. What's the standard method to allow a directory of files to be include()'d or otherwise read by PHP, but not be browsable from the general public?

(I think this says a lot about my need to move from just coding, and actually managing these darn servers.)

So far I disable access from .htaccess. Is there some standard method I'm missing, though? I'm not sure if permissions would work, because PHP and apache should run as the same user.

[KnightExemplar]

More sanity-checking. What's the standard method to allow a directory of files to be include()'d or otherwise read by PHP, but not be browsable from the general public?

(I think this says a lot about my need to move from just coding, and actually managing these darn servers.)

You put them in an earlier directory. For example:

/my_project_dir
.../www_public/ <--- Configure Apache to have this as directory_root
.../includes/ <--- Apache can't read this, because its not in directory_root. But you can read it in PHP with just

Code: Select all

include("../includes/blah")

I think anyway. You might be forced to use absolute paths instead of relative paths, its been a while. If so, then "realpath" is all you need to convert relative paths into absolute paths.

So far I disable access from .htaccess. Is there some standard method I'm missing, though? I'm not sure if permissions would work, because PHP and apache should run as the same user.

And also no. You don't need to have apache and php to run as the same user. By using php-fpm and Apache's FCGI module, you can even split the application server from the web server entirely, and run PHP on an entirely separate box. Putting PHP onto a different box (Ex: a different EC2 instance from Amazon) will allow for horizontal scalability, especially with a load balancing scheme. (Ex: multiple boxes running php independently while load balance using the client's IP address so that the client will typically use the same machine.)

You probably don't have a use of it yet, but its good to keep it in mind.

[phlip]

Slightly cleaner is to do that, and then put the appropriate path in your include_path, so you can just include() or require() them without the full path.

Another option is to do away with making the included files inaccessible, but instead put something like

Code: Select all

if (!defined('FROMMAIN'))
{
  header("HTTP/1.1 403 Forbidden");
  die('Nope.');
}

at the top of your includes, and then

Code: Select all

if ($_SERVER['SCRIPT_FILENAME'] == __FILE__) // Sanity check that we haven't been included in another file
  define('FROMMAIN', 1);

in your actual main code files that you expect people to be accessing. Then if they go to the included files directly, they get presented with an error.

I've seen both methods used, in different projects. The first method has the advantage that you don't have to worry about missing a file - you only have to set it up once, while the second method you need to add code to every single file. Whereas with the second one you have all your files in one place, but with the first you have files spread out across the filesystem (making the second one easier for packages that you intend for people to download and use themselves, since it's all self-contained).

[Steax]

Thanks guys. I think I need to do this for more than PHP files - say, cache files. If there's a cache of a restricted page, I don't want the cache file to be easily accessible. I think I'll go with KnightExemplar's first method, and drop them elsewhere. I'll certainly keep the other methods in mind. I swear I've used phlip's second one before, but apparently I've totally forgotten about it.

Second sanity check - if I wanted to store all files securely somewhere, but allow for specific users to download them at certain times, should use a temporary folder to copy the requested file into, and then later delete said file after the download completes? Or should I just have PHP read it as an octet-stream, and offer it as a download? I've had trouble getting download managers to work successfully on both mechanisms. It's annoyingly tricky, and I get the impression there's a better way to do this.

[Ubik]

I was going to answer that I would probably just use PHP's readfile(), but when I checked the documentation of the function, I noticed the topmost comment which mentions mod_xsendfile which seems relevant. Readfile or something similar is probably the most "portable" and safest (less likely to cause permission problems etc.) way of serving files - the copying thing sounds like the directory could fill up with stuff unless cleaned up often enough, and you can't restrict access to the files in as controlled fashion as you could when passing the file through a script.

[You, sir, name?]

I'm working on maintaining a system that's due for replacement fairly soon, basically trying to keep it together as long as possible before it's replacement is up to spec.

This isn't made easier by the fact that the guy previously maintaining it was a total git. He doesn't understand version control. He uses it as FTP. When he's made a significant change to a file, he makes a backup of the old version and uploads it to SVN with some suffix (like foo.java_old). And branching is way out of his league. He did all development in the main trunk.

And that isn't even talking about some of the code this guy has produced. Take basically the last 500 snippets of code posted to thedailywtf and string them together, and you get the code base I'm working with.

[TheChewanater]

And that isn't even talking about some of the code this guy has produced. Take basically the last 500 snippets of code posted to thedailywtf and string them together, and you get the code base I'm working with.

I assume that includes this?

[You, sir, name?]

And that isn't even talking about some of the code this guy has produced. Take basically the last 500 snippets of code posted to thedailywtf and string them together, and you get the code base I'm working with.

I assume that includes this?

I wouldn't put it past this guy.

He's the mastermind behind a class that looked roughly like this:

Code: Select all


public static final int FOO1 = 0;
public static final int FOO2 = 1;
...
public static final int FOO25 = 25;

int foo1() {
  return table[FooClass.FOO1];
}

int foo2() {
  return table[FooClass.FOO2];
}

...

int foo25() {
  return table[FooClass.FOO25];
}


The numbers in the function names and enumerations are not mine, but actually in the code.

table wasn't initialized in the constructor by the way, but in a public function called init() which was invoked by the constructor (and running it again would seriously mess up the sanity of the application).

[Xeio]

Fire the refactor cannon!

Refactor cannon is super effective!

Unecessary class fainted.

[elminster]

Over the last 2 days I recoded the graphics and input subsystems on the game I'm working on from legacy directx 7 DirectDraw to SDL. SDL is quite nicely wrapped for basic graphics and it took very little to recode it. Bit sparse on the documentation, but does the job. However, annoyingly, there's only a handful of blending modes.
It's got binds into OpenGL, so I might just do it using that, but was hoping I didn't need to.

[Sean Quixote]

Quicky question on cryptography:

So, I know little to nothing about it, though it's one of the higher-ups on my long list of intellectual curiosities. I'd just like to quickly expose an idea that I've always had to scrutiny from people more knowledgeable/intelligent than me:

I'm told that basically every encryption is crackable, all that's required is time and knowhow, and a good algorithm preferably requires lots of both. But what about stacked or nested algorithms? That is, say instead of relying on one single algorithm that you consider a "good" one, why not use MANY? Then, a hacker would be faced with multiple layers of encryption, and hell you could even have an algorithm that assigns a random number to represent all of your algorithms and then yet another to pick another random number that picks which algorithm is the one that's actually hiding your encrypted information. Then, it at least seems to me that, if you have enough separate algorithms at your disposal then the whole prospect of just choosing which algorithm to try to crack, becomes a mathematical clusterfuck of impossibility.

This is the first time I've ever actually written (and thus fully thought) out this idea, and another thought finally ocurrs to me. Either I'm right, and implementing such a system would be a complete mindfuck for hackers to deal with, or just the other way around: it would be a complete mindfuck to code in the first place and therefore impossible to implement.

[Robert'); DROP TABLE *;]

I'm told that basically every encryption is crackable, all that's required is time and knowhow, and a good algorithm preferably requires lots of both.

This is only true in theory; although all new vulnerabilities may be discovered in the future, modern high-security algorithms would require astronomical amounts of time and space to crack.

[Sean Quixote]

Another thought that occurred to me while I was typing that up but forgot somewhere along the way was the possibility that someone else before me has already thought this up, and that it's in fact already a widely-used practice.

But, the basic idea behind it originated from the fact that I'm always coming up with ideas for codes (or at least fledgling ideas that I rarely ever flesh out), usually pretty simple ones. But if you had, say, thousands or even millions of them, no matter how simple, then wouldn't the problem of deciding which one to crack be a whole other at least fairly complex problem in itself? I suppose at some point, memory would become the main issue. But I also like to think that that would be the beauty of the KISS method for your oodles of possible algorithms...

I need to stop fucking around and learn how to code, already. =\

[TheChewanater]

But if you had, say, thousands or even millions of them, no matter how simple, then wouldn't the problem of deciding which one to crack be a whole other at least fairly complex problem in itself?

Alternatively, adding three bytes to the key also multiplies the possibilities by over a million.

[lalop]

Also a layman here, but I think I might know the answer to these:

Another thought that occurred to me while I was typing that up but forgot somewhere along the way was the possibility that someone else before me has already thought this up, and that it's in fact already a widely-used practice.

But, the basic idea behind it originated from the fact that I'm always coming up with ideas for codes (or at least fledgling ideas that I rarely ever flesh out), usually pretty simple ones. But if you had, say, thousands or even millions of them, no matter how simple, then wouldn't the problem of deciding which one to crack be a whole other at least fairly complex problem in itself? I suppose at some point, memory would become the main issue. But I also like to think that that would be the beauty of the KISS method for your oodles of possible algorithms...

The notion here is that "security by obscurity" is not true security: you have to plan against the attackers knowing your algorithm, because stuff is always leaked, reverse-engineered, etc.

Even if you stack several algorithms on top of each other, there's still a chance that your process would get found out, so that's part of why your previous proposal isn't considered solid. Another part is that, the more algorithms you decide to stack, the more computationally expensive it is to encode or decode compared to just having a single "good" algorithm to start out with.

[headprogrammingczar]

Also, it's very easy to do cryptographic analysis on one among potential thousands of terrible encryption algorithms.

[b.i.o]

But, the basic idea behind it originated from the fact that I'm always coming up with ideas for codes (or at least fledgling ideas that I rarely ever flesh out), usually pretty simple ones. But if you had, say, thousands or even millions of them, no matter how simple, then wouldn't the problem of deciding which one to crack be a whole other at least fairly complex problem in itself? I suppose at some point, memory would become the main issue. But I also like to think that that would be the beauty of the KISS method for your oodles of possible algorithms...

As has been explained, security through obscurity is bad practice. One good encryption mechanism is a lot better than thousands or millions of bad ones. Most "simple" encryption mechanisms you're likely to think up are going to be vulnerable to simple attacks like frequency analysis.

And even if it was a good idea, all you've really done is made the encryption mechanism itself part of your key, and that key is symmetric, meaning the person you're communicating with needs to know all of the mechanisms too, and needs to know which you're using somehow. And if you have a secure way of getting the details of millions of encryption mechanisms to someone, you could probably just deliver enough one-time pads to last you a lifetime instead.

[Steax]

Alarmingly, I had a coworker a few weeks back who protested heavily against our multi-salted, 1000-times hashed password mechanism. He insisted, instead, on a straightforward MD5 with no salts. Decreases the chance of something going wrong, he says.

[PM 2Ring]

Alarmingly, I had a coworker a few weeks back who protested heavily against our multi-salted, 1000-times hashed password mechanism. He insisted, instead, on a straightforward MD5 with no salts. Decreases the chance of something going wrong, he says.

I think that your coworker is now obliged to prove that he is not
A) Intending to hack into the system.
B) An idiot.

[Steax]

We effectively removed him from the dev team since he used base64 in exchange for proper encryption to store critical user data.

"What do you mean, the plus and equal symbols would give it away to any competent programmer?"

[Thesh]

Quicky question on cryptography:

So, I know little to nothing about it, though it's one of the higher-ups on my long list of intellectual curiosities. I'd just like to quickly expose an idea that I've always had to scrutiny from people more knowledgeable/intelligent than me:

I'm told that basically every encryption is crackable, all that's required is time and knowhow, and a good algorithm preferably requires lots of both. But what about stacked or nested algorithms? That is, say instead of relying on one single algorithm that you consider a "good" one, why not use MANY? Then, a hacker would be faced with multiple layers of encryption, and hell you could even have an algorithm that assigns a random number to represent all of your algorithms and then yet another to pick another random number that picks which algorithm is the one that's actually hiding your encrypted information. Then, it at least seems to me that, if you have enough separate algorithms at your disposal then the whole prospect of just choosing which algorithm to try to crack, becomes a mathematical clusterfuck of impossibility.

This is the first time I've ever actually written (and thus fully thought) out this idea, and another thought finally ocurrs to me. Either I'm right, and implementing such a system would be a complete mindfuck for hackers to deal with, or just the other way around: it would be a complete mindfuck to code in the first place and therefore impossible to implement.

All block ciphers are basically multiple rounds of a simple cipher. Let's take serpent as an example, it does an XOR, substitution using 4x4 bit s-boxes (essentially 16 element arrays of all integers from 0-15), and then a linear transformation so that each output bit is dependent on each input bit. Do this once, cryptanalysis is trivial. However, after 32 rounds of this it is strong (the best known attack on serpent only breaks 12 rounds, but is still infeasible).

If you were going to combine a bunch of weak algorithms, you should just choose a bunch of algorithms and run them in the same order no matter what. This simplifies cryptanalysis which allows security to be proven against current techniques. Remember, it's only strong against modern techniques if the best cryptanalysts can't break it. Adding too much randomness and no one can analyze the entire cipher and you run into the possibility of weak keys. Also, you need to use a good key stretching algorithm so that each cipher gets a unique subkey.

That said, keep in mind that cryptanalysts will not be cracking each cipher individually, but cracking the cipher as a whole. You can do this just by looking at inputs and outputs. So you do need to put thought into the choice of your algorithms to minimize various characteristics that a cryptanalyst can use.

I think a more interesting technique is to use multiple strong ciphers, alternating stream and block ciphers that have been well studied and have not been broken. Most attacks use known or chosen plaintexts, and if you encrypt with a stream cipher and then a block cipher (even encrypting in ECB mode it is fine if it is already encrypted with a stream cipher) it makes those attacks significantly more difficult.

[Pepve]

[...] multi-salted, 1000-times hashed password mechanism [...]

I don't understand this, can you elaborate?

[Windowlicker]

Minor change of subject: I'm working on a project in Python, and want to implement my own tab auto-completion. I think I should be able to handle it mostly, the only thing I'm unsure about is whether or not I can edit the text that the user currently has typed in. That is, how do I change the input from "doc" to "documents" from within the code? Hopefully I'm getting my point across... I'd just google it, but I can't think of a phrase to get back only relevant results.

[You, sir, name?]

Minor change of subject: I'm working on a project in Python, and want to implement my own tab auto-completion. I think I should be able to handle it mostly, the only thing I'm unsure about is whether or not I can edit the text that the user currently has typed in. That is, how do I change the input from "doc" to "documents" from within the code? Hopefully I'm getting my point across... I'd just google it, but I can't think of a phrase to get back only relevant results.

http://docs.python.org/library/readline.html

[Windowlicker]

I thank you sir.

[b.i.o]

[...] multi-salted, 1000-times hashed password mechanism [...]

I don't understand this, can you elaborate?

The right way to store passwords is to not store the actual password, but to store a hashed version of the password. When a user goes to log in, you hash their password again, and compare the hashes. If they're the same, the user almost certainly entered the correct password. This is better than storing the password in plaintext, because (as long as a good cryptographic hash function is used) it's very hard or impossible to get the original password back from the hashed version.

One problem with this scheme is that it's relatively easy to compute rainbow tables for common hash schemes by just running the hash functions on lots of common words, which gives you a mapping from hash back to password. Users choose bad passwords, so this is a problem. One way to avoid it is to add a salt to the hash: a long, random string that gets stored in plaintext. Even if the attacker knows the salt, it prevents precomputed rainbow tables from being effective. (And if you use 1 salt per user, which you absolutely should, it forces an attacker to try to compute a separate rainbow table for each individual user.)

However, there's still a pretty large problem with this: for many other uses, people want their hash functions to be fast. And if a hash can be computed fast enough, then it's possible for an attacker to recover the passwords of users who have using short or otherwise bad passwords. We're pretty much at the point where using a single run of a general-purpose hash function like SHA-1 is a bad idea. (MD5 is bad for this and other, worse reasons.)

The solution is to slow the hashing computation down. One way to do this is to use a hash function that's specifically designed to be slow like bcrypt or scrypt. Both have a parameter that allows you to adjust how computationally difficult hashing is. Another thing to do is just to iterate the hash function (keep re-hashing the hashed value) a lot of times (1000+), which effectively slows down an attacker by three orders of magnitude.

[Pepve]

Thank you for the explanation. What about the "multi-salted" bit?

[Aaeriele]

Thank you for the explanation. What about the "multi-salted" bit?

Multi-salted means using more than one salt value - for instance, if you're iterating 1000 times with 10 salts, you might use the [(n mod 10)+1]th salt value for each iteration.

[b.i.o]

Multi-salted means using more than one salt value - for instance, if you're iterating 1000 times with 10 salts, you might use the [(n mod 10)+1]th salt value for each iteration.

Why would you do this? The point of hashing 1000 times is that the computation takes longer. Adding extra salts in doesn't appear to add any value that I can see. Am I missing something?

[Jplus]

(Excuse me for ignoring the hash/salt discussion.)

I've been going through A Tour of Go. I used to be very sceptical about Go ("oh, yet another high level language"), but today I heard it was designed by Rob Pike so I got curious and I went through the tutorial. To my surprise it's really a modern reincarnation of C.

[headprogrammingczar]

Actually... It's a modern reincarnation of Algol 68.

[Steax]

Multi-salted means using more than one salt value - for instance, if you're iterating 1000 times with 10 salts, you might use the [(n mod 10)+1]th salt value for each iteration.

Why would you do this? The point of hashing 1000 times is that the computation takes longer. Adding extra salts in doesn't appear to add any value that I can see. Am I missing something?

It increases the necessary amount of time if anyone seriously wants to get past the hashing.

Say it takes t amount of time to generate a full rainbow table for a particular hashing algorithm.
Adding a salt: 1t (just adding a string to each key)
Adding 2 salts: 2t
Adding n salts: nt

If a single salt was used, the time it would take to create a full rainbow table would be equal to that of no salt - just append the salt. If you use just two salts, you double that time. 10 salts, 10 times the time. It's also quite straightforward to make, as it's just repeating over a number of prefixes and suffixes. If extra security is necessary, you could even use both a prefix and suffix array, each with a prime amount of members - say, 13 and 17. This would increase the time necessary to build a rainbow table by 200 times, so at this point it's probably more practical to take, say, the $2 approach.

[Thesh]

The time to calculate the rainbow table (or, in this case, use a brute force or dictionary attack) is proportional to the number of iterations, not the number of salts. The only thing a salt accomplishes is preventing the use of pre-computed rainbow tables. Beyond that, it adds nothing.

[Steax]

The time to calculate the rainbow table (or, in this case, use a brute force or dictionary attack) is proportional to the number of iterations, not the number of salts. The only thing a salt accomplishes is preventing the use of pre-computed rainbow tables. Beyond that, it adds nothing.

Assuming an attacker is aware of our multiple-hashing function and our use of salts (this being the worst-case scenario where the attacker knows our hashing algorithm):

A set of 1000 repeats without a salt only requires computing a rainbow table for each possibility (16^160 for, say, SHA-1), and resulting in another hash. Since each is just a rehash, this single table can be looked up to traverse any number of repeats.

If you use a single salt, they just need to make a table of those 16^160 strings, plus the salt. This doesn't increase the time necessary to build said table.

If you use two salts, they need to make a table of 16^160 + salt strings, twice. This means twice the time required.


Of course, this is assuming our attacker has the power and will to attempt building rainbow tables. Most probably won't. However, the additional effort on the programming bit is negligible (using simple math to figure out which salt to use where), so I don't see why not.