Python & GIL

larsp (4) skrev:

Der er vel to sider af sagen. Hvad der er korrekt Python kode, og hvad der forventes af C / C++ koden for native kode moduler. Jeg tænker at den største udfordring er at porte alverdens native kode moduler væk fra at bruge GIL?

This PEP poses a number of backwards compatibility issues when building CPython with the --disable-gil flag, but those issues do not occur when using the default build configuration. Nearly all the backwards compatibility concerns involve the C-API:

CPython builds without the GIL will not be ABI compatible with the standard CPython build or with the stable ABI due to changes to the Python object header needed to support biased reference counting. C-API extensions will need to be rebuilt specifically for this version.
C-API extensions that rely on the GIL to protect global state or object state in C code will need additional explicit locking to remain thread-safe when run without the GIL.
C-API extensions that use borrowed references in ways that are not safe without the GIL will need to use the equivalent new APIs that return non-borrowed references. Note that only some uses of borrowed references are a concern; only references to objects that might be freed by other threads pose an issue.

larsp (4) skrev:

Kommer der mon til at være forskelle hvad angår acceptabel Python kode? Jeg har altid undgået at tilgå de samme lists og dictionaries, f.eks., fra forskellige tråde samtidigt, uden at have undersøgt om man egentlig godt kunne nu når der er en GIL. Går det mon fra at være acceptabelt til ulovligt?

arne_v (3) skrev:

#2

Python 2 -> 3 var en breaking change. Der var masser af python kode som skulle ændres. Det har det med at forsinke opdatering.

(Om GIL: ...) This simplifies the CPython implementation by making the object model (including critical built-in types such as dict) implicitly safe against concurrent access (...)

Do not rely on the atomicity of built-in types.

While Python's built-in data types such as dictionaries appear to have atomic operations, there are corner cases where they aren't atomic (e.g. if __hash__ or __eq__ are implemented as Python methods) and their atomicity should not be relied upon. Neither should you rely on atomic variable assignment (since this in turn depends on dictionaries).

larsp (7) skrev:

#6 Hvad gjorde de bedre i Swift og de andre sprog "udviklet af virksomheder" for at håndtere breaking changes mellem versioner?

Claus Jørgensen (10) skrev:

#8

volatile gør ikke en variable thread safe. Bare fordi compileren ikke cache din variable, gør det ikke den threadsafe.

Claus Jørgensen (9) skrev:

Automated tooling til opgradering

An object that has volatile-qualified type may be modified in ways unknown to the
implementation or have other unknown side effects. Therefore any expression referring
to such an object shall be evaluated strictly according to the rules of the abstract machine,
as described in 5.1.2.3. Furthermore, at every sequence point the value last stored in the
object shall agree with that prescribed by the abstract machine, except as modified by the
unknown factors mentioned previously. 134) What constitutes an access to an object that
has volatile-qualified type is implementation-defined.

Unless I am mistaken, the only reason FreeRTOS doesn’t fall apart during untimely context switches is because most platforms which run it are single-core microprocessors where a write to an address is not buffered or cached in some way. In these cases, volatile has been “good enough” to stop breaking optimizations. However, this is a very poor safety net.

larsp (17) skrev:

#volatile

Jeg researchede emnet lidt nærmere. Claus har ret i at man, i dag, aldrig bør stole på at en variabel er threadsafe i C/C++. Volatile ændrer ikke på dette. Man bør bruge enten specielle atomic angivelser for den givne compiler, eller C11 "_Atomic" eller C++11 "std::atomic".

Det er ganske relevant at have atomiske variable i high performance kode eller systemprogrammering. Hvis man kan lave en data struktur lock-free er det ofte en fordel. Et typisk eksempel er en ringbuffer som nemt kan laves lock-free med atomiske og volatile head og tail indeks variable.

Ved mere simple CPUer plejede det at være "godt nok" at bruge en volatile int. Men den går ikke længere. FreeRTOS har egg-on-the-face i denne sammenhæng! https://www.freertos.org/FreeRTOS_Support_Forum_Ar...

Unless I am mistaken, the only reason FreeRTOS doesn’t fall apart during untimely context switches is because most platforms which run it are single-core microprocessors where a write to an address is not buffered or cached in some way. In these cases, volatile has been “good enough” to stop breaking optimizations. However, this is a very poor safety net.

Der er også andre eksempler, f.eks. gode gamle Atmel AVR, som er udbredt i Arduinoer. Denne processor er 8-bit og har 8-bit registre, men de valgte at gøre "int" størrelsen 16-bit i AVR-GCC, så en "volatile int" er absolut ikke atomisk her. En "volatile int8_t" er tilgengæld sikker.

larsp (17) skrev:

#16 En CriticalSection er (i microcontroller verdenen) en billig måde at opnå atomicitet. Man slår simpelthen interrupts fra og til rundt om operationen. Jeg ved ikke præcis hvad en critical section gør i f.eks. Windows. Holder den f.eks. andre tråde i en multikerne CPU væk? Det kan vel kun gøres ved at stoppe alle relevante tråde i denne section. Ikke så billigt længere!

En Mutex er et noget mere kompliceret dyr, der kan have fancy features som priority inheritance (på et RTOS).

arne_v (19) skrev:

En Win32 critical section eller POSIX mutex virker lidt ligesom Java synchronized og C# lock.

Kun en tråd af gangen kan eje den critical section / mutex.

Og det bruges til at beskytte en shared resource.

Andre tråde der ikke skal bruge den resource kører full speed.

Tråde der skal bruge den resource er nødt til at vente indtil de kan få den.

Dyrt?

Der er overhead ved at have tråde der skal vente på en anden tråd.

Men overhead ved at bruge en critical section / mutex til at få dem til at vente er minimalt.

Så det dyre er i designet ikke i synkroniserings mekanismen.

class SomeActor {
receive loop {
msg = receive_message()
switch(msg) {
... => ...
... => ...
... => ...
}
}
}

public class TemperatureLogger
{
private List<int> measurements;
private int max;
public TemperatureLogger(int measurement)
{
measurements = new List<int>();
measurements.Add(measurement);
max = measurement;
}
public int GetMax()
{
return max;
}
public void Update(int measurement)
{
measurements.Add(measurement);
if(measurement > max)
{
max = measurement;
}
}
}

public class TemperatureLogger : ReceiveActor
{
public enum Command
{
GetMax,
Stop
}
public class Update
{
public int Measurement { get; set; }
}
private List<int> measurements;
private int max;
public TemperatureLogger(int measurement)
{
measurements = new List<int>();
measurements.Add(measurement);
max = measurement;
Receive<Command>((Command cmd) => {
switch (cmd)
{
case Command.GetMax:
Sender.Tell(max);
break;
case Command.Stop:
Self.GracefulStop(TimeSpan.FromMilliseconds(100));
break;
}

});
Receive<Update>((Update updmsg) => {
measurements.Add(updmsg.Measurement);
if (updmsg.Measurement > max)
{
max = updmsg.Measurement;
}
});

}
}

public class TemperatureLogger
{
private List<int> measurements;
private int max;
public TemperatureLogger(int measurement)
{
measurements = new List<int>();
measurements.Add(measurement);
max = measurement;
}
public int GetMax()
{
lock(this)
{
return max;
}
}
public void Update(int measurement)
{
lock(this)
{
measurements.Add(measurement);
if(measurement > max)
{
max = measurement;
}
}
}
}